Minimalistic file uploader
Hi dude. It’s me. I mean, you’re from the future. Alas, here in 2023 we don’t have any flying cars and skateboards. And the funny thing is that transferring files between devices is still a problem. I hope you read this and create a better timeline for yourself.
In the meantime, I’m stuck here and have to somehow upload pictures from my phone, which for some reason lost its MTP. I have a feedback page for a completely static site in the works and I thought – Oh! But there the file loader will be very handy, at the same time I will throw off the pictures. And as soon as I started doing it, I see a message in one of the Devuan mailing lists: how did they get these fucking phones, how to upload files, please help.
Well, I think, if I’m not the only one, then it’s worth it. I used to do something like this, a long time ago, back in the days of jQuery, but then there was some kind of ready-made component. Now, I didn’t want anything else. Useful in MDN. Meaningfully copied and pasted everything, and here I am reporting the results. Of course, I cleaned up some things, for example, keeping track of files with the same name if they are from different directories, showing thumbnails, but this is solely for the sake of simplicity. You can easily do these little things yourself.
So, the basic Javascript loader module. You can call it a module with a big stretch, since all Javascript and CSS I have is collected in one HTML file, packed into gz, and then nginx distributes it to the left and right as quickly as possible. Inside the HTML, the Javascript module becomes anonymous without the ability to export anything, so you have to use the bad old methods.
(() => {
class FileUploader
{
constructor(settings)
{
const default_settings = {
url: '/',
chunk_size: 512 * 1024, // последний chunk может быть в полтора раза больше,
// не забываем про лимиты request body на сервере
// (у NGINX по умолчанию 1M)
file_name_header: 'File-Name' // что-нибудь стандартное типа Content-Disposition
// было бы лучше, но его сложнее парсить
};
this.settings = Object.assign({}, default_settings, settings);
this.upload_queue = [];
}
upload(file, params)
/*
* Добавляем файл в очередь, и если загрузка ещё не в процессе - тогда начинаем.
*/
{
const start_upload = this.upload_queue.length == 0;
// Создаём file_item и добавляем в начало очереди
const file_item = new FileItem(this, file, params);
this.upload_queue.push(file_item);
if(start_upload) {
// если вызываем асинхронную функцию без await, получим promise,
// либо fulfilled, либо pending. Но он нам всё равно не нужен.
this._async_upload_files().then();
}
}
progress(file, params, chunk_start_percentage, chunk_end_percentage, percentage)
/*
* Этот метод вызывается для отображения прогресс-бара.
* Реализуем его в производном классе.
*/
{
}
async upload_complete(file, params)
/*
* Этот метод вызывается по завершении загрузки.
* Реализуем его в производном классе.
*/
{
}
async _async_upload_files()
{
// обрабатываем очередь загрузки
while(this.upload_queue.length != 0) {
await this.upload_queue[0].upload();
this.upload_queue.shift();
}
}
}
class FileItem
/*
* Элемент очереди загрузки.
*/
{
constructor(uploader, file, params)
{
this.uploader = uploader;
this.file = file;
this.params = params;
}
async upload()
{
var chunk_start = 0;
var chunk_size;
while(chunk_start < this.file.size) {
const remaining_size = this.file.size - chunk_start;
// загружаем кусками default_chunk_size, последний кусок допускается
// в полтора раза больше, чем default_chunk_size
if(remaining_size < 1.5 * this.uploader.settings.chunk_size) {
chunk_size = remaining_size;
} else {
chunk_size = this.uploader.settings.chunk_size;
}
const chunk = this.file.slice(chunk_start, chunk_start + chunk_size);
// XXX сохранять (start, end) в слайсе - грязный хак, а что делать?
chunk.start = chunk_start;
chunk.end = chunk_start + chunk_size;
while(true) {
try {
await this._upload_chunk(chunk);
break;
} catch(error) {
console.log(`${this.file.name} upload error, retry in 5 seconds`);
await new Promise(resolve => setTimeout(resolve, 5000));
}
}
chunk_start += chunk_size;
}
await this.uploader.upload_complete(this.file, this.params);
}
_upload_chunk(chunk)
{
// Эта функция использует non-awaitable XMLHttpRequest, поэтому не может быть async.
// Но мы вызываем её с await, так что должны вернуть promise.
const self = this;
return new Promise((resolve, reject) => {
const reader = new FileReader();
const xhr = new XMLHttpRequest();
xhr.upload.addEventListener(
"progress",
(e) => {
if(e.lengthComputable) {
const percentage = Math.round((e.loaded * 100) / e.total);
self._update_progress(chunk, percentage);
}
},
false
);
xhr.onreadystatechange = () => {
if(xhr.readyState === xhr.DONE) {
if(xhr.status === 200) {
self._update_progress(chunk, 100);
resolve(xhr.response);
} else {
reject({
status: xhr.status,
statusText: xhr.statusText
});
}
}
};
xhr.onerror = () => {
reject({
status: xhr.status,
statusText: xhr.statusText
});
};
xhr.open('POST', this.uploader.settings.url);
const content_range = `bytes ${chunk.start}-${chunk.end - 1}/${this.file.size}`;
xhr.setRequestHeader("Content-Range", content_range);
xhr.setRequestHeader("Content-Type", "application/octet-stream");
xhr.setRequestHeader(this.uploader.settings.file_name_header, this.file.name);
reader.onload = (e) => {
xhr.send(e.target.result);
};
reader.readAsArrayBuffer(chunk);
self._update_progress(chunk, 0);
});
}
_update_progress(chunk, percentage)
{
// считаем проценты и вызываем метод progress
const chunk_start_percentage = chunk.start * 100 / this.file.size;
const chunk_end_percentage = chunk.end * 100 / this.file.size;
const upload_percentage = chunk_start_percentage + chunk.size * percentage / this.file.size;
this.uploader.progress(
this.file,
this.params,
chunk_start_percentage.toFixed(2),
chunk_end_percentage.toFixed(2),
upload_percentage.toFixed(2)
);
}
}
// типа экспортируем FileUploader
window.FileUploader = FileUploader;
})();HTML and rest of Javascript:
<h3>Upload Files</h3>
<p>
<button id="file-select">Choose Files</button> or drag and drop to the table below
</p>
<table id="file-list">
<thead>
<tr><th>File name</th><th>Size</th></tr>
</thead>
<tbody>
</tbody>
</table>
<template id="file-row">
<tr><td></td><td></td></tr>
</template>
<input type="file" id="files-input" multiple style="display:none">
<script>
const upload_complete_color="rgb(0,192,0,0.2)";
const chunk_complete_color="rgb(0,255,0,0.1)";
class Uploader extends FileUploader
{
constructor()
{
super({url: '/api/feedback/upload'});
this.elem = {
file_select: document.getElementById("file-select"),
files_input: document.getElementById("files-input"),
file_list: document.getElementById("file-list"),
row_template: document.getElementById('file-row')
};
this.elem.tbody = this.elem.file_list.getElementsByTagName('tbody')[0];
this.row_index = 0;
this.set_event_handlers();
}
set_event_handlers()
{
const self = this;
this.elem.file_select.addEventListener(
"click",
() => { self.elem.files_input.click(); },
false
);
this.elem.files_input.addEventListener(
"change",
() => { self.handle_files(self.elem.files_input.files) },
false
);
function consume_event(e)
{
e.stopPropagation();
e.preventDefault();
}
function drop(e)
{
consume_event(e);
self.handle_files(e.dataTransfer.files);
}
this.elem.file_list.addEventListener("dragenter", consume_event, false);
this.elem.file_list.addEventListener("dragover", consume_event, false);
this.elem.file_list.addEventListener("drop", drop, false);
}
progress(file, params, chunk_start_percentage, chunk_end_percentage, percentage)
{
params.progress_container.style.background = 'linear-gradient(to right, '
+ `${upload_complete_color} 0 ${percentage}%, `
+ `${chunk_complete_color} ${percentage}% ${chunk_end_percentage}%, `
+ `transparent ${chunk_end_percentage}%)`;
}
async upload_complete(file, params)
{
// красим зелёным всю строку
params.progress_container.style.background = upload_complete_color;
params.progress_container.nextSibling.style.background = upload_complete_color;
}
handle_files(files)
/*
* обрабатываем здесь файлы от drag'n'drop или диалога выбора
*/
{
for(const file of files) {
const cols = this.append_file(file.size);
this.upload(file, {progress_container: cols[0]});
}
}
append_file(size)
/*
* Добавляем файл в таблицу, возвращаем список ячеек.
*/
{
const rows = this.elem.tbody.getElementsByTagName("tr");
var row;
if(this.row_index >= rows.length) {
row = this.append_row();
} else {
row = rows[this.row_index];
}
this.row_index++;
const cols = row.getElementsByTagName("td");
cols[1].textContent = size.toString();
return cols;
}
append_row()
/*
* Добавляем пустую строку к таблице.
*/
{
const tbody = this.elem.file_list.getElementsByTagName('tbody')[0];
const row = this.elem.row_template.content.firstElementChild.cloneNode(true);
tbody.appendChild(row);
return row;
}
const uploader = new Uploader();
// инициализируем таблицу - добавляем пять пустых строк
for(let i = 0; i < 5; i++) uploader.append_row();
</script>And finally, a piece of the server side:
import os.path
import re
from starlette.responses import Response
import aiofiles.os
# Ничего этого в aiofiles нет. На момент написания, по крайней мере.
aiofiles.os.open = aiofiles.os.wrap(os.open)
aiofiles.os.close = aiofiles.os.wrap(os.close)
aiofiles.os.lseek = aiofiles.os.wrap(os.lseek)
aiofiles.os.write = aiofiles.os.wrap(os.write)
re_content_range = re.compile(r'bytes\s+(\d+)-(\d+)/(\d+)')
@expose(methods="POST")
async def upload(self, request):
'''
Ловим и записываем кусок файла.
'''
data = await request.body()
filename = os.path.basename(request.headers['File-Name'])
start, end, size = [int(n) for n in re_content_range.search(request.headers['Content-Range']).groups()]
fd = await aiofiles.os.open(filename, os.O_CREAT | os.O_RDWR, mode=0o666)
try:
await aiofiles.os.lseek(fd, start, os.SEEK_SET)
await aiofiles.os.write(fd, data)
finally:
await aiofiles.os.close(fd)
return Response()What to say in conclusion? We’re running in circles. The processors are more powerful, there is more memory, and I could not download all the files at once. The browser crashed with OOM after 20-30 uploaded photos, and that’s without showing thumbnails. Or did I screw up somewhere?
