ptpb/pb

building pb from a non-git copy

coolph007 opened this issue · 14 comments

coolph007 commented

Hello,
I'm facing below error while trying to run docker-compose up command.

https://pasteboard.co/IfLRsvx.jpg

Could you please help me.

buhman commented

That error happens because you are building from a non-git source, which is not supported for the build method you are attempting. This is fully explained with relevant github-specific information in the message.

Make sure you clone pb with git clone https://github.com/ptpb/pb.

buhman commented

Alernately, if your issue is you don't have git for some reason, you can use this dockerfile instead of the one that ships in the repo:

FROM python:3.6-alpine

WORKDIR /pb

RUN apk add --no-cache --virtual .build-deps git
RUN git clone https://github.com/ptpb/pb /build
RUN pip install /build
RUN apk del .build-deps

CMD ["python", "-m", "pb"]
glensc commented

@buhman version creates layer with buld deps. combine apk add + apk del to same RUN statement, or use multi stage builds.

FROM python:3.6-alpine

WORKDIR /pb

RUN set -x \
  && apk add --no-cache --virtual .build-deps git \
  && git clone https://github.com/ptpb/pb /build \
  && pip install /build \
  && apk del .build-deps

CMD ["python", "-m", "pb"]
FROM python:3.6-alpine

RUN apk add --no-cache git
RUN git clone https://github.com/ptpb/pb /build
RUN pip install /build

FROM python:3.6-alpine

COPY --from=0 /usr/local /usr/local
CMD ["python", "-m", "pb"]

the multi stage may have other dependencies you may need to copy, so it's not safest to have. you need to verify outcome with every change.

  • 👍1
coolph007 commented

That error happens because you are building from a non-git source, which is not supported for the build method you are attempting. This is fully explained with relevant github-specific information in the message.

Make sure you clone pb with git clone https://github.com/ptpb/pb.

Thank you so much. Works now.

However, I'm not finding any way to go to back to the terminal to execute commands. Ctrl+C disconnects the process. Sorry, I'm new to linux.

https://pasteboard.co/IfTqHWk.jpg

Also, could you tell me where the pastes are stored? In mongodb or unix file system? Will pastes persist if I restart the docker container?

buhman commented

However, I'm not finding any way to go to back to the terminal to execute commands

There are several possible answers to this question.

  • If you want to re-use the same terminal for (local) commands, you can background docker-compose with docker-compose up --detach.
  • If you want to execute a command in a running container's context, you can use docker exec. List running containers with docker ps, and run a shell inside, say, the pb app container with docker exec -ti pb_pb_1 /bin/sh (this also matches the name shown on the left column of docker-compose run output.

If this doesn't answer your question, please clarify which commands you're interested in running.

In mongodb or unix file system?

Never on the filesystem. Pastes are stored entirely mongodb (both metadata and content), sometimes via GridFS. My original idea was mongo offers reasonably good horizontal scaling and sharding out of the box, so these properties would apply to pb as well. If I were to write pb again from scratch, I would use filesystem apis instead.

Ctrl+C disconnects the process. Sorry, I'm new to linux.

I suggest reading about Ctrl-Z, bg, fg and other job control commands. https://www.digitalocean.com/community/tutorials/how-to-use-bash-s-job-control-to-manage-foreground-and-background-processes appears to be a fairly concise explanation.

Will pastes persist if I restart the docker container?

Pastes will persist. You can control this behavior with --no-recreate and --force-recreate accordingly. Keep in mind there is a strict terminology difference between container and container image. Restarting a process in a container is conceptually/literally the same thing as restarting a process anywhere else. When you create a new container from a new container image, the new container has no knowledge of and is completely separated from data that might exist elsewhere (this is by design).

  • 👍1
buhman commented

@glensc indeed, thanks for reminding me!

coolph007 commented

Thanks for your detailed reply.

Pastes are stored entirely mongodb

Is there any way I can limit the size of the pastes that the API can accept (say less than 1 MB or so) and is it possible to accept pastes only from a specific host?

buhman commented

Is there any way I can limit the size of the pastes that the API can accept (say less than 1 MB or so)
and is it possible to accept pastes only from a specific host?

Both of these can be done with nginx.

http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-tcp/

You could look at https://github.com/ptpb/ptpb-deploy/blob/11b6485f55a77d8328ca96d5295cb187f7797a29/openresty/ptpb.conf#L18-L43 for a good starting point; ignore lines 26-28.

coolph007 commented

@buhman - Thank you.

coolph007 commented

Pastes will persist. You can control this behavior with --no-recreate and --force-recreate accordingly. Keep in mind there is a strict terminology difference between container and container image. Restarting a process in a container is conceptually/literally the same thing as restarting a process anywhere else. When you create a new container from a new container image, the new container has no knowledge of and is completely separated from data that might exist elsewhere (this is by design).

Sorry I'm not really familiar with docker terminology. Is there any way I can ensure that all pastes persists if something unexpected happens? How can I take backup of mongodb?

Thanks.

buhman commented

How can I take backup of mongodb?

You'll want to use "docker volumes". This is explained in the context of mongo in https://hub.docker.com/_/mongo

For example,

docker run -v /my/own/datadir:/data/db -d mongo

Would start a container using the mongo image, with /my/own/datadir on the host mapped to /data/db in the container. /data/db is where the mongo image stores its data by default.

In docker-compose, this would be:

  mongodb:
    image: mongo:latest
    volumes:
      - '/local/db/path:/data/db'
buhman commented

How can I take backup of mongodb?

A backup can be a cp/rsync of the local /data/db, as long as mongod is stopped while the copy operation is in progress. https://docs.mongodb.com/manual/core/backups/ also suggests using filesystem snapshots to avoid this offline requirement.

A third alternative is to use mongodump as also described there, but I suggested filesystem-based backups as mongodump/restore are somewhat (albeit bearably) slow if you have a huge database such as ptpb.pw's. You would use docker exec to start mongodump inside the container.

coolph007 commented

Thank you for the detailed response.

glensc commented

as for mongo backup, you may want to use my tool: https://github.com/glensc/docker-mongodb-hotbackup

but it may require you to use percona version of mongodb to have hotbackup support

  • 👀1