UCSB-VRL/bisqueUCSB

TurboGears==2.2.1

Closed this issue · 13 comments

amilworks commented

TurboGears 2.2.1

This is the second upgrade in the process. If you have any questions, tag @amilworks on here.
Current Version: TurboGears2==2.2.0
Upgrade Version: TurboGears2==2.2.1

As you are going through, check off what Step you completed to make it easier for us to help, and for you to feel accomplished.

Getting Started

Step 1. Clone the Repo

GOAL

  • Start by cloning the branch from this current repo
git clone -b tg-upgrade --single-branch https://github.com/UCSB-VRL/bisqueUCSB.git

cd tg-upgrade and you should see a folder structure like this

.
├── boot
├── builder
├── COPYRIGHT
├── DOCKER-BUILD_LOG.md
├── Dockerfile.caffe.xenial
├── DOCKER-RUN_LOG.md
├── entry.sh
├── LICENSE
├── Makefile
├── modules
├── README.md
├── requirements.txt
├── run-bisque.sh
├── source
├── sources.list
├── start-bisque.sh
└── virtualenv.sh

4 directories, 13 files

Step 2. Verify Everything Works

GOAL

  • Establish a baseline that BisQue works on your local system before changing anything.

You have two choices to verify that the repo is working.

Option 1. Pull the Latest Docker Build (Recommended)

The repo is rebuilt and pushed automatically after a successful push/merge on GitHub. Download the latest image by running the following command:

docker pull amilworks/bisque-tg2-upgrade:git

You can run this image to verify BisQue is working and able to run locally on your system by running:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

You should see BisQue on http://localhost:8080/ in your browser. If not, paste the logs below and we will take a look.

Option 2. Build the Image

Building the image from scratch is what you will end up doing throughout the upgrade process. Luckily, we automated it to where you run a Makefile.

make bisque05-caffe-xenial

If you vim Makefile, you will see it's a series of basic commands. I suggest you Edit this file and --tag the image something different and update the version you are upgrading instead of amilworks/bisque05-caffe-xenial:flour-tg2.2. You can leave it but it's kind of long. Once the build is done, verify that BisQue comes up on your local system just as above:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

You should see BisQue on http://localhost:8080/ in your browser. If not, paste both the Build and Run Logs below and we will take a look.

Step 3. Make Changes/Break Stuff

GOAL

  • Upgrade TurboGears to TurboGears2==2.2.1, fix problems that arise, document changes.

Create a new branch
Creating a new branch will enable you to make changes, nuke your own branch over and over again, and maintain a clean copy of the original branch when you need to start fresh for the millionth time.

git checkout -b [name_of_your_new_branch]

Open source/requirements.txt with your favorite editor (hopefully Vim). Change the version to the one you are assigned, and save the file.

Here is the link to the line you will update.

TurboGears2==2.2   # <--- Update Version
TurboMail==3.0.3
WebError==0.10.3
WebFlash==0.1a9
WebHelpers==1.3
WebOb==1.1.1       # <--- Potential Update Needed, Check Build Logs

Push to GitHub
You made changes and you want to have version control. Push your new branch and changes to GitHub by running:

git push -u origin  [name_of_your_new_branch]

Step 4. Build and Run!

GOAL

  • Build and run your changes, fix any dependency issues, document changes, rebuild if necessary (most likely unless you are super lucky), trust the process, and always ask for help if you get stuck!

Once you made your changes, make sure you cd .. out of source/ and are back to the tree view displayed above. You should see a file called Makefile. You can Edit this file as mentioned in Option 2. or simply run the following command to build the image with your updated TurboGears2.

make bisque05-caffe-xenial

While its building, make sure that there are no errors during the pip installs. Typically errors are skipped over in favor of a successful build. I know, right?

Next, once the build is finished, run the image and verify BisQue comes up on your local system:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

NOTE: There might be other packages that need updating when upgrading to different TurboGears2 versions. Sometimes there will be an error raised with the version required, or it simply will not run and you will have to do a logsdump here and we will have to sift through it together to track the failed step(s).

If everything is successful, push your branch to GitHub if you have not already, file a pull request, and pat yourself on the back. Job well done. We will review it and test it on our production-like setup and give feedback if any issues arise.

Post Upgrade

Use this issue to document any issues you faced during the upgrade process. Log dumps can be thrown here as well. Try to use the code syntax for large blocks of code so we can easily read through it. Otherwise, go crazy.

  • What broke, if anything? Just a simple list will suffice.
  • Were there any files significantly changed or that requires significant attention?
  • Can we move forward to the next upgrade?
  • Any blockers now or that you foresee?
ASMIftekhar commented

I think this is not a serious issue but when I see bisque on my local browser it shows an error but the site comes up and all :

Error
Requested resource does not exist: /services

ASMIftekhar commented

I cant push local update to the repo,
remote: Permission to UCSB-VRL/bisqueUCSB.git denied to ASMIftekhar.

ASMIftekhar commented

Getting this error after changing TG version and running the make file:

Step 7/23 : RUN wget -q https://bitbucket.org/dimin/bioimageconvert/downloads/$IMGCNV.tar.gz && tar xf $IMGCNV.tar.gz && mv $IMGCNV/imgcnv /usr/bin && mv $IMGCNV/libimgcnv.so* /usr/lib/x86_64-linux-gnu/ && rm -rf $IMGCNV && apt-get install -y --no-install-recommends libswscale-ffmpeg3 libfftw3-3 libgdcm2.6 libavcodec-ffmpeg56 libavformat-ffmpeg56 libavutil-ffmpeg54 libhdf5-cpp-11
---> Running in 8b6c936f4981
The command '/bin/sh -c wget -q https://bitbucket.org/dimin/bioimageconvert/downloads/$IMGCNV.tar.gz && tar xf $IMGCNV.tar.gz && mv $IMGCNV/imgcnv /usr/bin && mv $IMGCNV/libimgcnv.so* /usr/lib/x86_64-linux-gnu/ && rm -rf $IMGCNV && apt-get install -y --no-install-recommends libswscale-ffmpeg3 libfftw3-3 libgdcm2.6 libavcodec-ffmpeg56 libavformat-ffmpeg56 libavutil-ffmpeg54 libhdf5-cpp-11' returned a non-zero code: 8
Makefile:9: recipe for target 'bisque05-caffe-xenial' failed
make: *** [bisque05-caffe-xenial] Error 8

ASMIftekhar commented

Changed TurboGear from 2.1.5 to 2.2.1. Make got well but running the container gives me the following error:

  • '[' -d /builder/boot-scripts.d ']'
  • ls -l /builder/boot-scripts.d
    total 4
    -rwxrwxr-x 1 root root 47 Dec 9 19:44 B10-fullconfig.sh
  • for f in '${BUILD}boot-scripts.d/B*.sh'
  • echo 'Executing **BOOT /builder/boot-scripts.d/B10-fullconfig.sh **'
    Executing **BOOT /builder/boot-scripts.d/B10-fullconfig.sh **
  • '[' -f /builder/boot-scripts.d/B10-fullconfig.sh ']'
  • bash /builder/boot-scripts.d/B10-fullconfig.sh
  • bq-admin setup -y fullconfig
    Traceback (most recent call last):
    File "/usr/lib/bisque/bin/bq-admin", line 6, in
    from pkg_resources import load_entry_point
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 3036, in
    @_call_aside
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 3020, in _call_aside
    f(*args, **kwargs)
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 3049, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 656, in _build_master
    return cls._build_from_requirements(requires)
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 669, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
    File "/usr/lib/bisque/local/lib/python2.7/site-packages/pkg_resources/init.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (WebOb 1.0.8 (/usr/lib/bisque/lib/python2.7/site-packages), Requirement.parse('WebOb==1.1.1'), set(['TurboGears2']))
ASMIftekhar commented

So I changed web0b==1.0.8 to 1.1.1 and build again. The built went well but after running the docker container I got this error.
This means something got broken due to the update of web0b. Any way to track this?
image

amilworks commented

So its saying that the Turbogears version you have installed needs WebOb to be upgraded. Nice work!

Requirement.parse('WebOb==1.1.1'), set(['TurboGears2']))
amilworks commented

Can you try upgrading the requests package?

ASMIftekhar commented

To which version?

ASMIftekhar commented

This comment is for @mcever @FeiX-OvO and @ivanfarevalo.
You first need to build the container by running:
make bisque05-caffe-xenial
Then docker run command should be:
docker run -itp 8080:8080 --name bqdev-tg2 amilworks/bisque05-caffe-xenial:flour-tg2.2

amilworks commented

So I tried to find a specific/relevant version, but couldn't find anything that was definitive like "Use this version for WebOb==1.1.1". So my suggestion is naive, let's try the latest since it supports Python 2.7. We will try this extreme, then we can try the opposite extreme of the next version bump from our current one.

pip install requests==2.25.0
ASMIftekhar commented

Didnt work, it broke some other package name MonekeyPatch. Are you sure Web0b is not working because of requests. Then may be we can try randomly different requests versions.

amilworks commented

Yeah, so you're right. WebOb actually doesn't have any dependencies, from what I can see, except for the Python Standard library. That's pretty crazy. This will probably require a little more digging because from the looks of it, nothing was Raised to say "upgrade package X to v2.x". Here are the changes from the commit log for turbogears2==2.2. I highlighted one in bold that might be the reason for the /services issue but I am not completely sure.

@satish1901 Do you have any ideas?

  • Fix template engine extension collision in tw2.core >= 2.1
  • Permit TGController subclasses to keep the decorations made on the parent class methods around
  • Support repoze.who v2
  • New auth configuration layer that doesn't require repoze.what anymore while still being compatible
  • Support validation dict using TW2 validators and make validation less dependant from FormEncode
  • Always prefer default renderer when multiple engines are registered for the same content_type
  • Fix empty documentation for some decorators
  • Remove url kwargs which were marked for removal in 2.2
  • Remove legacy renderers support
  • Disable AuthMetadataProvider when is None
  • Improved kajiki loader to solve some issues with py:extends
  • use .jinja extension for jinja templates by default
  • On Mako when using file loader templates where reloaded using the
    auto_reload_templates option, while when using dotted loader the
    mako.reloadfromdisk option was used. Merge the two options in
    auto_reload_templates as all the other templates engines use this
    one
  • before and after were renamed to _before and _after
  • Expose render call as a public api with render_template name
  • Create sqla module for sqlalchemy related things and add BalancedSession for master/slave load balancing
  • Change prefix for tw2 to make resources archiving work correctly
  • prefer_toscawidgets2 option in app_cfg enables tw2 and disables
    tw2. This should be the default in 2.2 newly quickstarted projects
  • Add option for setting the template rendering parameters.
  • Permit to avoid doctype injection when manually rendering templates with tg.render.render
  • Object dispatch has been delegated to the crank package, improving performance and test coverage
  • Quickstart new applications using repoze.who v2
  • Add dynamic response.charset to meta in mako templates
  • New quickstart template
  • Quickstart projects without repoze.what requirement using new auth layer
  • Fix issue with kajiki quickstart
  • Adapt the bootstrap based template to jinja changes, always provide
    master.html for extensions and remove the minimal template as the
    new one is minimal enough
  • Make genshi always available to have a lingua franca for pluggable apps and extensions
  • Storing sessions in encrypted cookies is easier to manage then the file based solution for newly quickstarted projects
  • tgext.admin now works with ToscaWidgets2
  • Paste quickstart command will now ask if you want to use an alternative templating system instead of asking for each one
  • New Upgrading TG2 page
  • Document new inherit option of @expose with examples
  • Fix missing kajiki setup_renderer reference and remove reference to default_renderer
github-actions commented

Stale issue message