brainsteam.co.uk/brainsteam/content/posts/legacy/2018-10-20-uploading-huge-f...

2.8 KiB
Raw Blame History

title author type date url medium_post categories tags
Uploading HUGE files to Gitea James post 2018-10-20T10:09:41+00:00 /2018/10/20/uploading-huge-files-to-gitea/
O:11:"Medium_Post":11:{s:16:"author_image_url";N;s:10:"author_url";N;s:11:"byline_name";N;s:12:"byline_email";N;s:10:"cross_link";s:2:"no";s:2:"id";N;s:21:"follower_notification";s:3:"yes";s:7:"license";s:19:"all-rights-reserved";s:14:"publication_id";s:2:"-1";s:6:"status";s:4:"none";s:3:"url";N;}
PhD
Work
devops
docker
git
lfs

I recently stumbled upon and fell in love with Gitea a lightweight self-hosted Github and Gitlab alternative written in the Go programming language. One of my favourite things about it other than the speed and efficiency that mean you can even run it on a raspberry pi is the built in LFS support. For the unfamiliar, LFS is a protocol initially introduced by GitHub that allows users to version control large binary files something that Git is traditionally pretty poor at.

Some of my projects have huge datasets that I want to store somewhere safe and keep under version control. LFS is not perfect but it is a reasonable solution for this particular problem.

When I installed Gitea I was initially disappointed that uploading large files to LFS seemed to result in errors. I was getting:

api error: Authentication required: Authorization error: <REPO_URL>/info/lfs/objects/batch
Check that you have proper access to the repository
batch response: Authentication required: Authorization error: <REPO_URL>/info/lfs/objects/batch
Check that you have proper access to the repository

Irritatingly I couldnt find any references to this particular error message or documentation. But, I had a hunch that the authentication on the LFS upload was timing out because I was able to upload smaller files that dont take as long to send.

It turns out that this is exactly what was happening. When you push to a Gitea SSH repository, the server gives your local machine an authorization token that it can use to upload the LFS files. This token has an expiry time which defaults to 20 minutes in the future. If youre uploading 5GB of data over 100Mb/down 10Mb/up DSL line then youre gonna have a bad time…

I had a dig through the Gitea github repository and came across an example config file which includes a variable called LFS_HTTP_AUTH_EXPIRY with a default value of 20m. In your gitea config file you can set this to 120m then you have 2 hours to get that file uploaded. Adjust as you see fit/require.