Batch Uploader tests « MediaDrop Community

MediaDrop Community » General Discussion & Feedback

Batch Uploader tests

19 posts Started 11 years ago by jsalgado Latest reply from Kessir

jsalgado
Member
Posted 11 years ago

I think we all would really apreciate a faster automated way to upload videos.

I have been testing with 3 aproaches:

a ) mcoreuploader: A simple Upload API client from mediacore.com
b ) curl POST to "http://mediacoreip/upload/submit"
c ) Local File upload

My Results:

a) mcoreuploader (the Client upload API from mediacore.com)
seems to be quite compatible but no joy

Stuck on "Max retries exceeded for url: /api/uploader/media"

more info: https://mediacore.com/developer-platform/mcoreuploader#example

Has any body got theese uploader scripts to work with the mediacore-community version?

b) Curl POST
At the beginning it was just a desperate measure, because its not a good aproach at all, but I needed to upload 200+ videos to a new site.

No joy either, file gets uploaded but the server rejects it every time.
My personal script:
#curl -F file=@/home/NewProject.mp4 -F name=jsalgado -F email=jsalgado%40uno.com -F title=Test1 -F description=Testing http://mymediacorei-ip.tv/upload/submit

more info: http://journalxtra.com/linuxsanity/bash-linuxsanity/bash-filling-web-forms-with-curl-and-wget-3814/

c) The local file upload
I copied all my files to the mediacore server, but I Have to run a browser on the server to "upload" files that are already there!

A slightly better option was to publish an FTP server, on the same server and link ftp urls instead of uploading

Results: works but its not an improvement, its just faster.

Any ideas?
jsalgado
Member
Posted 11 years ago

mcoreuploader output:

# python MediaCoreUploader-1.1c/mcupload.py

Traceback (most recent call last):
File "MediaCoreUploader-1.1c/mcupload.py", line 7, in <module>
media = client.create_media(title='TEST1')
File "/home/ubuntu/MediaCoreUploader-1.1c/mcoreuploader/client.py", line 116, in create_media
raise MCoreConnectionError(str(e))
mcoreuploader.exceptions.MCoreConnectionError: Max retries exceeded for url: /api/uploader/media

# ---------------- Script listing -----------
# MediaCoreUploader-1.1c/mcupload.py
# -------------------------------------------

from mcoreuploader import Client

client = Client(url='ventana.dyndns.tv',
username='jsalgado',
password='mypassword')

media = client.create_media(title='TEST1')

media.upload_file('/home/ubuntu/test.mp4')

------------------ script end -----------------
jsalgado
Member
Posted 11 years ago

CURL output

#curl -F file=@/home/ubuntu/NewProject.mp4 -F name=jsalgado -F email=jsalgado%40uno.com%0d%0a -F title=prueba -F description=prueba http://mymediacore-ip.tv/upload/submit > respuesta.html

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 712k 100 172 100 712k 17 74390 0:00:10 0:00:09 0:00:01 64968

(the file gets uploaded but its rejected)

# tail -f /var/log/nginx/error.log

2013/02/15 09:16:56 [error] 5637#0: *44 upstream prematurely closed connection while reading response header from upstream, client: 189.163.255.72, server: ventana.dyndns.tv, request: "POST /upload/submit HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi-mediacore.sock:", host: "ventana.dyndns.tv"

# respuesta.html on my browser

502 Bad Gateway
nginx/1.0.5
am262327
Member
Posted 11 years ago

I think the best approach is to use a python script:

from mediacore.lib.storage import add_new_media_file

The add_new_media_file is actually defined in mediacore/lib/storage/api.py. If I were you I would just try it out and use something like IPython or maybe just print statements to see how you do.

I quite agree; batch export and import is essential for any tool.
jsalgado
Member
Posted 11 years ago

Thank you very much am262327, I did take a look at mediacore/lib/storage/api.py and the developer documents, but either I lack python skills, or the documentation is a little short.

I haven't dare to test "add_new_media_file" because I feared to mess up my database. And because the upload is a two step process. I need to first create the media "field" and then upload the actual "video file"

Does any body has an "add_new_media_file" example to share with us?
am262327
Member
Posted 11 years ago

Yes, it does require a bit of python skills. Actually, I think I may have led you on the wrong track because mediacore.controllers.upload.save_media_obj looks to be much closer to what you need. It creates the media object from the given parameters, and then calls add_new_media_file itself.

But this isn't a function that you can just use from a python intepretor though. This is a bit tricky to execute because I'm pretty sure (I haven't actually tried it) that you'll have to have the app running for this to work.

I'm actually kind of interested in getting this going, but I don't have the time to look into it now. I think there should be a setting in the admin side that imports and exports movies.

And yes, you always want to test things on a clean database that you don't care about messing up. :)
Felix Schwarz
MediaDrop developer
Posted 11 years ago

I think the best way forward is a read/write JSON api, maybe with some example tools to import media. I have already two customers who are interested in that but unfortunately their budget doesn't yet cover my estimated cost.

My idea is to support exactly the same API as MediaCore (cloud edition) + some useful extensions maybe. If someone is interested in joining (either as developer or spending some money) let me know (info@schwarz.eu).

Other than that I have an older command-line import. It is separated in two pieces and you would need to write the part which gathers the media information (e.g. file system location, title, ...) and write that into a JSON file. The second part then puts that into the MediaCore DB.
valerio.minetti
Member
Posted 11 years ago

Ok, just to recap: mediacore ce does not have a rest api for bulk uploads? does the mcoreuploader works or it is just for the cloud edition?
Felix Schwarz
MediaDrop developer
Posted 11 years ago

Yes, MediaCore CE does not offer an upload API yet so the mcoreuploader won't work (as that is just the client part, you could even use PHP or Java for that). Implementing the REST api is planned for MediaCore CE 0.11.

Good news is that I recently found a plugin (https://github.com/catturavideo/mediacore-ce-upload-plugin) which should provide an upload API. So far I neither reviewed the code nor tried it but I'd be really interested in your experience trying it.
jsalgado
Member
Posted 11 years ago

Hi, I finally solved my problem developing my own shell upload code. I have to use php since I am more experienced using php.

Its not a plug-and-play solution, but it was a life saver for me. I uploaded the video files to a folder on my server using scp / winscp , then the php script creates the 3 thumbnails sizes, uploads fields to the database and publishes all videos whith their thumnails.

If anyone is interested I can generalize the script and upload it to share.
Felix Schwarz
MediaDrop developer
Posted 11 years ago

I think it would be great if you would publish your code. For example in a blog post. If you don't have a blog I could also publish your post on the official "blog".

If there is some interest from others/the code is being improved over time eventuall there should be a repository but I'm a big fan of publishing example code in a blog post and see what happens.
valerio.minetti
Member
Posted 11 years ago

Hi Felix,
if you or other are interested i've created a tiny php daemon for local encoding with ffmpeg. It scans mediacore db for unencoded media, transcode them into mp4 and then add media_files entries for the transcoded version.
At the end of the process it sets media as encoded and send an email to media author.

It's not polished yet since it has no encoding presets management ( i just used one to see if things where working fine) but it seems to do its job well.

I also included an upstart script in order to daemonize it.
If someone want to give it a try i've created a public repo on github:

https://github.com/valeriominetti/mediacore-ffmpeg-bot

Note: i didn't write the code for tumbnail and media creation because i meant this object just like a transcoder helper and i wanted my user to start the workflow loading their files and filling the media metadata.

Note2: i wrote this in php since i'm pretty a novice with python and had a strict timeline for this task. since it's completly decoupled from mediacore i felt safe doing it this way :)
Felix Schwarz
MediaDrop developer
Posted 11 years ago

Valerio: That's great! Feel free to announce that on the developer mailing list. Also if you are interested in writing a blog post (Markdown format), I'm happy to publish that on the community site. Great work.

Btw: If you had posted this in a new thread probably you would get more visibility.
Felix Schwarz
MediaDrop developer
Posted 11 years ago

Btw: I published a user survey to learn about your priorities with regard to new features.
valerio.minetti
Member
Posted 11 years ago

Hi,
with developer mailing list do ya mean the mediacoreCE group on google groups? anyway thanks for the suggestion, i'll open a new thread too.
Felix Schwarz
MediaDrop developer
Posted 11 years ago

exactly (MediaCore mailing list).
amine963
Member
Posted 10 years ago

Hello,
here is my old very basic uploader to mediacore engine: http://pastebin.com/6Fz1WYni

It's not complete, since it lacks librairies, very quick'n'dirty, but you can grab snippets from it.

Basically, it connects to a mysql database to retrieve data, then instanciate a "Client" object to populate Mediacore database.

Sorry for my english and the poor quality of this code, very old.
renecalles
Member
Posted 10 years ago

Hi Felix,

how about a plugin which observes a specified folder (e.g. (S)FTP) for new incoming media? This could also be an an option then for large video transferring. So, User uploads to an FTP => MediaCore finds the newly added item and creates a Media Item according to this Media File.

As extension there could be an XML specifying the metadata for this Media Item.

What do you think?

Best,
René
Kessir
Member
Posted 10 years ago

Hi jsalgado do you mind sharing your php code for batch upload?

RSS feed for this topic

Reply

You must log in to post.

Community Forums

Batch Uploader tests

Reply