Uploads errored when processed


#1

Setup : 0.17 with docker

The tracks are uploaded correctly, but their processing is pending.

In celeryworker logs, I have the following error :

celeryworker_1  | 2018-12-05T12:56:15.349442200Z [2018-12-05 12:56:15,348: INFO/MainProcess] Received task: music.process_upload[587caea8-c89a-49f7-8eb5-06cf43ba1be2]  
celeryworker_1  | 2018-12-05T12:56:16.652424005Z [2018-12-05 12:56:16,635: WARNING/ForkPoolWorker-2] [celery] Error during task 587caea8-c89a-49f7-8eb5-06cf43ba1be2: [Errno 2] No such file or directory: '/srv/funkwhale/data/media/tracks/18/56/46/bB3_Sagittarius.flac'
celeryworker_1  | 2018-12-05T12:56:16.654100646Z [2018-12-05 12:56:16,653: WARNING/ForkPoolWorker-2] Traceback (most recent call last):
celeryworker_1  | 2018-12-05T12:56:16.655149315Z [2018-12-05 12:56:16,654: WARNING/ForkPoolWorker-2] File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
celeryworker_1  | 2018-12-05T12:56:16.655372811Z     R = retval = fun(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.656046536Z [2018-12-05 12:56:16,655: WARNING/ForkPoolWorker-2] File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
celeryworker_1  | 2018-12-05T12:56:16.656260720Z     return self.run(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.656913614Z [2018-12-05 12:56:16,656: WARNING/ForkPoolWorker-2] File "/app/funkwhale_api/taskapp/celery.py", line 51, in inner
celeryworker_1  | 2018-12-05T12:56:16.657124162Z     return function(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.658025019Z [2018-12-05 12:56:16,657: WARNING/ForkPoolWorker-2] File "/app/funkwhale_api/music/tasks.py", line 179, in process_upload
celeryworker_1  | 2018-12-05T12:56:16.658228978Z     audio_file = upload.get_audio_file()
celeryworker_1  | 2018-12-05T12:56:16.660395905Z [2018-12-05 12:56:16,658: WARNING/ForkPoolWorker-2] File "/app/funkwhale_api/music/models.py", line 703, in get_audio_file
celeryworker_1  | 2018-12-05T12:56:16.660685650Z     return self.audio_file.open()
celeryworker_1  | 2018-12-05T12:56:16.661698537Z [2018-12-05 12:56:16,661: WARNING/ForkPoolWorker-2] File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 76, in open
celeryworker_1  | 2018-12-05T12:56:16.662010698Z     self.file = self.storage.open(self.name, mode)
celeryworker_1  | 2018-12-05T12:56:16.663099879Z [2018-12-05 12:56:16,662: WARNING/ForkPoolWorker-2] File "/usr/local/lib/python3.6/site-packages/django/core/files/storage.py", line 33, in open
celeryworker_1  | 2018-12-05T12:56:16.663361258Z     return self._open(name, mode)
celeryworker_1  | 2018-12-05T12:56:16.667605508Z [2018-12-05 12:56:16,663: WARNING/ForkPoolWorker-2] File "/usr/local/lib/python3.6/site-packages/django/core/files/storage.py", line 218, in _open
celeryworker_1  | 2018-12-05T12:56:16.667842791Z     return File(open(self.path(name), mode))
celeryworker_1  | 2018-12-05T12:56:16.668815707Z [2018-12-05 12:56:16,668: WARNING/ForkPoolWorker-2] FileNotFoundError: [Errno 2] No such file or directory: '/srv/funkwhale/data/media/tracks/18/56/46/bB3_Sagittarius.flac'
celeryworker_1  | 2018-12-05T12:56:16.670581493Z [2018-12-05 12:56:16,669: ERROR/ForkPoolWorker-2] Task music.process_upload[587caea8-c89a-49f7-8eb5-06cf43ba1be2] raised unexpected: FileNotFoundError(2, 'No such file or directory')
celeryworker_1  | 2018-12-05T12:56:16.670641036Z Traceback (most recent call last):
celeryworker_1  | 2018-12-05T12:56:16.670665744Z   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
celeryworker_1  | 2018-12-05T12:56:16.670689371Z     R = retval = fun(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.670712123Z   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
celeryworker_1  | 2018-12-05T12:56:16.670735690Z     return self.run(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.670758190Z   File "/app/funkwhale_api/taskapp/celery.py", line 51, in inner
celeryworker_1  | 2018-12-05T12:56:16.670781469Z     return function(*args, **kwargs)
celeryworker_1  | 2018-12-05T12:56:16.670804497Z   File "/app/funkwhale_api/music/tasks.py", line 179, in process_upload
celeryworker_1  | 2018-12-05T12:56:16.670827524Z     audio_file = upload.get_audio_file()
celeryworker_1  | 2018-12-05T12:56:16.670850216Z   File "/app/funkwhale_api/music/models.py", line 703, in get_audio_file
celeryworker_1  | 2018-12-05T12:56:16.670873555Z     return self.audio_file.open()
celeryworker_1  | 2018-12-05T12:56:16.670896751Z   File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 76, in open
celeryworker_1  | 2018-12-05T12:56:16.670920306Z     self.file = self.storage.open(self.name, mode)
celeryworker_1  | 2018-12-05T12:56:16.670942878Z   File "/usr/local/lib/python3.6/site-packages/django/core/files/storage.py", line 33, in open
celeryworker_1  | 2018-12-05T12:56:16.670966409Z     return self._open(name, mode)
celeryworker_1  | 2018-12-05T12:56:16.670989029Z   File "/usr/local/lib/python3.6/site-packages/django/core/files/storage.py", line 218, in _open
celeryworker_1  | 2018-12-05T12:56:16.671012320Z     return File(open(self.path(name), mode))
celeryworker_1  | 2018-12-05T12:56:16.672771747Z FileNotFoundError: [Errno 2] No such file or directory: '/srv/funkwhale/data/media/tracks/18/56/46/bB3_Sagittarius.flac'

And the track folder structure :

tracks
└── 2018
    β”œβ”€β”€ 07
    β”‚   β”œβ”€β”€ 10
    β”‚   β”‚   └── imports
    β”‚   β”‚       └── 2018
    β”‚   β”‚           └── 07
    β”‚   β”‚               └── 10
    β”‚   β”œβ”€β”€ 11
    β”‚   β”‚   └── imports
    β”‚   β”‚       └── 2018
    β”‚   β”‚           └── 07
    β”‚   β”‚               └── 11
...

#2

I suspect you’re bit by https://docs.funkwhale.audio/changelog.html#ensure-media-root-static-root-and-music-directory-are-set-explicitely-manual-action-required

Can you please ensure you have https://code.eliotberriot.com/funkwhale/funkwhale/blob/develop/deploy/docker-compose.yml#L38 correctly set for the api and celeryworker containers ?


#3

I’ve changed my docker-compose.yml to match the current one.

Upon restarting the container, I have the following error :

nginx_1         | 2018-12-05T17:50:45.259052027Z envsubst: error while reading "standard input": Is a directory

Is it safe to down -v the nginx container ?
As a side note, some research on this error brought up :

If you are having this issue your volume path is most likely wrong. Make sure you if you are using relative paths that you have ./ in front and double check the path exists locally.

So I checked the paths in the .env :

FUNKWHALE_VERSION=0.17
MUSIC_DIRECTORY_PATH=/music
MUSIC_DIRECTORY_SERVE_PATH=/srv/funkwhale/data/music
FUNKWHALE_FRONTEND_PATH=/srv/funkwhale/front/dist
MEDIA_ROOT=/srv/funkwhale/data/media
STATIC_ROOT=/srv/funkwhale/data/static

#4

I think the issue lies in the nginx volumes: can you check that all nginx volumes point to actual files (and not directories ?


#5

I think so. Here is the nginx block :

  nginx:
    image: nginx
    env_file:
      - .env
    environment:
      # Override those variables in your .env file if needed
      - "NGINX_MAX_BODY_SIZE=${NGINX_MAX_BODY_SIZE-30M}"
    volumes:
      - "./nginx/funkwhale.template:/etc/nginx/conf.d/funkwhale.template:ro"
      - "./nginx/funkwhale_proxy.conf:/etc/nginx/funkwhale_proxy.conf:ro"
      - "${MUSIC_DIRECTORY_SERVE_PATH-/srv/funkwhale/data/music}:${MUSIC_DIRECTORY_SERVE_PATH-/srv/funkwhale/data/music}:ro"
      - "${MEDIA_ROOT}:${MEDIA_ROOT}:ro"
      - "${STATIC_ROOT}:${STATIC_ROOT}:ro"
      - "${FUNKWHALE_FRONTEND_PATH}:/frontend:ro"
    ports:
      # override those variables in your .env file if needed
      - "${FUNKWHALE_API_IP}:${FUNKWHALE_API_PORT}:80"
    command: >
        sh -c "envsubst \"`env | awk -F = '{printf \" $$%s\", $$1}'`\"
        < /etc/nginx/conf.d/funkwhale.template
        > /etc/nginx/conf.d/default.conf
        && cat /etc/nginx/conf.d/default.conf
        && nginx -g 'daemon off;'"
    links:
      - api

#6

The issue is likely with ./nginx/funkwhale.template, does the file exist on disc, and what is the content?


#7

Nice catch :slight_smile: What gave you the hint ?


#8

It’s a typical issue with docker volumes. When you’re running a container with a volume file like this docker run -v /tmp/test.txt:/test.txt debian,Docker will create and mount the given path as a directory if it does not exist on disK :smiley:


#9

Now uploads are working just fine, and federation is working just as fine. Fantastic. Thanks a lot :slight_smile:

Still have upload issues on my raspberry non docker instance, but this is definitely a win !