Ripping an album from youtube - CLI Style
With the advent of Spotify, Apple Music, Youtube, Pandora and many other streaming music services, the need to have local mp3 files doesn't crop up very often. However, my kids either have cheap mp3 players or use their 3ds's to play local mp3 files.
This post is a quick tip on ripping an album from youtube using a web browser and a few cli apps. Remember, most tasks don't need a bloated gui to be done efficiently.
Requirement
- A Web browser that can play youtube videos
- Youtube-dl
- ffmpeg
- Bash
Prep work
Install ffmpeg
Ubuntu
sudo apt-get install ffmpeg -y
Fedora
sudo yum install ffmpeg
Gentoo
sudo emerge ffmpeg
Install Youtube-DL
If your on a Debian or Ubuntu flavor of linux
sudo apt-get install youtube-dl -y
Fedora
sudo yum install youtub-dl
On my favourite, Gentoo
pip3 install youtube-dl --user
Download the album
At this point you have all the tools you need to get the job done. Have a browse around on youtube to find the album you want an offline copy of and copy the url of the page. Then from a command prompt:
mkdir ~/tmp
cd ~/tmp
youtube-dl -x --audio-format mp3 https://youtube.com/fullurltovideo
Create a list file
While the audio file is downloading, your going to want to create a simple txt file which lists the tracks, titles and start and end timings. I simply fast forwarded through each track toward the end of the song and made note of the mintes and seconds. I created a file with each line representing a track in the album with the following details:
Track Number-Track title-Start duration-End duration
The durations are in the form of HH:MM:SS. Here is what my file looks like:
cat ~/tmp/list.txt
01-The Greatest Show-00:0:00-5:08
02-A Million Dreams-00:5:08-9:38
03-A Million Dreams Reprise-00:9:39-10:38
04-Come Alive-00:10:38-14:25
05-The Other Side-00:14:25-17:58
06-Never Enough-00:17:58-21:28
07-This Is Me-00:21:38-25:23
08-Rewrite the Stars-00:25:23-28:59
09-Tightrope-00:28:59-32:50
10-Never Enough (Reprise)-00:32:50-34:14
11-From Now On-00:34:14-40:12
Split the audio to seperate mp3's
Now that my file has finished downloading, I can convert the file into seperate song files Here is the little bash script I wrote to split the file based on the contents of the list.txt file
cat splitsong.sh
#!/bin/bash
while read -r p; do
TRACK=$(echo "$p" | awk -F- '{print $1}')
TITLE=$(echo "$p" | awk -F- '{print $2}')
START=$(echo "$p" | awk -F- '{print $3}')
END=$(echo "$p" | awk -F- '{print $4}')
FILENAME="${TRACK} - The Greatest Showman - ${TITLE}.mp3"
ffmpeg -i "$2" -acodec copy -ss "$START" -to "$END" "${FILENAME}" < /dev/null
done <"$1"
I then execute the file like this:
chmod +x splitsong.sh
./splitsong.sh list.txt 'Some Sound Track List-qDZLSHY1ims.mp3'
And the whole thing is over in a matter of seconds.
Tags: bash-tips, mp3, ffmpeg, cli, script, youtube, music, linux
How I Code
Updated 17/08/2018
Coding can be fun. I've enjoyed coding from a young age, starting with GW-Basic at maybe 6, 7, or 8.
I remember my brother Alex seemed like a real genius with the computer (an IBM clone made by Acer 8086 XT). Using Basic he could make the computer do anything and was writing his own games.
Back then, editing code we would laugh at today and we take the humble text editor for granted. Take this example, you would type list to see your code:
>LIST
10 PRINT "WELCOME TO JESSES GAME"
20 PRINT "ENTER YOUR NAME"
30 $I = INPUT
40 PRINT "WELCOME $I, STRAP YOURSELF IN"
To edit a line of code you would re-write it by typing it in, line number and all.
20 PRINT "ENTER YOUR FULL NAME"
And to insert a line, start a line with a number between existing lines
31 $A=$I
When you ran out of in-between-lines there was a command you could run to reindex your lines which would space them all out 10 between each other.
Anyhow, when my Dad was about the same age as I am now (35), he went back to University to study Computer Science. I remember him bringing home Slackware and RedHat on floppies, which we would install and he would give me lessons on using Vi possibly vim, but I didn't know at the time. (This is probably around 1996).
Since finishing School and entering the workforce I have mostly worked in Windows environments. Even still, with the occasionaly need to touch GNU/Linux at work and often testing Distro's at home I would always feel more efficient when using Vi/m.
My feeling when using another editor is that moving around and changing text feels so lethargic when done one button at a time. This drove me in recent years to keep a copy in my home profile.
Around 2011 I switched from VBScript and the occasional perl script to writing fulltime in Powershell, so it made sense to try a few different editors which are more native to the Windows platform. I tried Visual Studio Code, Powershell ISE, Notepad++ and still kept coming back to vim.
Visual Studio Code is a great alternative, and it's Powershell extensions are very good. Hoever being an electron app, it suffers from performance and memory consumption issues. I love squeezing every drop of battery out of my PC and when you see 7mb RAM on Vim vs 500Mb+ on VSCode, you might rethink your choices.
Therefore I've resorted to delving into the world of customizing vim and setting up plugins.
One of the main things I'm trying to acheive is a cross platform configuration. You see, at work I'm on Windows and MacOs and at home I'm on Gentoo Linux. So I have written my .vimrc file to work on any platform. I usually sync it with OneDrive for Business and symlink it into my linux/mac/Windows home directory with a seperate setup script. Without further ado, here it is with some comments
.vimrc
if has("win32") " Check if on windows \
" Also supports has(unix)
source $VIMRUNTIME/mswin.vim " Load a special vimscript \
" ctrl+c and ctrl+v support
behave mswin " Like above
set ff=dos " Set file format to dos
let os='win' " Set os var to win
set noeol " Don't add an extra line \
" at the end of each file
set nofixeol " Disable the fixeol : Not \
" not sure why this is needed
set backupdir=~/_vimtmp,. " Set backupdir rather \
" than leaving backup files \
" all over the fs
set directory=~/_vimtmp,. " Set dir for swp files \
" rather than leaving files \
" all over the fs
set undodir=$USERPROFILE/vimfiles/VIM_UNDO_FILES " Set persistent undo\
" files
" directory
let plug='$USERPROFILE/.vim' " Setup a var used later to \
" store plugins
set shell=powershell " Set shell to powershell \
" on windows
set shellcmdflag=-command " Arg for powrshell to run
else
set backupdir=~/.vimtmp,.
set directory=~/.vimtmp,.
set undodir=$HOME/.vim/VIM_UNDO_FILES
let uname = system('uname') " Check variant of Unix \
" running. Linux|Macos
if uname =~ "Darwin" " If MacOS
let plug='~/.vim'
let os='mac' " Set os var to mac
else
if isdirectory('/mnt/c/Users/jpharris')
let plug='/mnt/c/Users/jpharris/.vim'
let os='wsl'
else
let plug='~/.vim'
let os='lin'
endif
endif
endif
execute "source " . plug . "/autoload/plug.vim"
if exists('*plug#begin')
call plug#begin(plug . '/plugged') " Enable the following plugins
Plug 'tpope/vim-fugitive'
Plug 'junegunn/gv.vim'
Plug 'junegunn/vim-easy-align'
Plug 'jiangmiao/auto-pairs'
"Plug 'vim-airline/vim-airline' " Airline disabled for perf
Plug 'morhetz/gruvbox'
Plug 'ervandew/supertab'
Plug 'tomtom/tlib_vim'
Plug 'MarcWeber/vim-addon-mw-utils'
Plug 'PProvost/vim-ps1'
Plug 'garbas/vim-snipmate'
Plug 'honza/vim-snippets'
call plug#end()
endif
" Remove menu bars
if has("gui_running") " Options for gvim only
set guioptions -=m " Disable menubar
set guioptions -=T " Disable Status bar
set lines=50 " Set default of lines
set columns=80 " Set default of columns
if os =~ "lin"
set guifont=Fira\ Code\ 12
elseif os =~ "mac"
set guifont=FiraCode-Retina:h14
else
set guifont=Fira_Code_Retina:h12:cANSI:qDRAFT
set renderoptions=type:directx
set encoding=utf-8
endif
set background=dark
colorscheme gruvbox
else
set mouse=a
if has('termguicolors')
set termguicolors " Enable termguicolors for \
" consoles which support 256.
set background=dark
colorscheme gruvbox
endif
endif
if has("persistent_undo")
set undofile " Enable persistent undo
endif
colorscheme evening " Set the default colorscheme
" Attempt to start vim-plug
syntax on " Enable syntax highlighting
filetype plugin indent on " Enable plugin based auto \
" indent
set tabstop=4 " show existing tab with 4 \
" spaces width
set shiftwidth=4 " when indenting with '>', \
" use 4 spaces width
set expandtab " On pressing tab, insert 4 \
" spaces
set number " Show line numbers
" Map F5 to python.exe %=current file
nnoremap <silent> <F5> :!clear;python %<CR>
" Remap tab to auto complete
imap <C-@> <C-Space>
" Setup ga shortcut for easyaline in visual mode
nmap ga <Plug>(EasyAlign)
" Setup ga shortcut for easyaline in normal mode
xmap ga <Plug>(EasyAlign)"
Tags: vim, coding, windows, linux, macos
Burning a DVD Video on Gentoo
Quick note for my future self
Overview
- Convert media to dvd compatible format
- Author DVD title
- Author DVD Table of Contents
- Convert DVD folder to ISO
- (Optional) Loopback mount ISO and test.
- Burn ISO to DVD
Packages Required
media-video/ffmpeg
media-video/dvdauthor
app-cdr/dvd+rw-tools
Commands
Start by using ffmpeg to convert the media to a dvd compatible format:
ffmpeg -i Big\ Buck\ Bunny.mp4 -target pal-dvd BigBuckBunny.mpg
Now use dvdauthor to author a title
dvdauthor -t -o dvd --video=pal -f BigBuckBunny.mpg
Add a table of contents
dvdauthor -T -o dvd
Create the ISO file
mkisofs -dvd-video -o BigBuckBunny.iso dvd/
(Optional) Mount to a loopback for testing
mkdir mount
mount -o loop BigBuckBunny.iso mount/
Play the video using VLC or some other tool to check it, then unmount
umount mount/
Burn to a disc
growisofs -dvd-compat -Z /dev/sr0=BigBuckBunny.iso
Credit to andrew.46 over at the ubuntuforums
Tags: burn-a-dvd, gentoo, ffmpeg, linux
BTRFS on Raspberry Pi
This weekend sees a change to my 2TB attached storage on the Raspberry Pi. I've been using this very Pi your reading this page from as Media server, Local file share, Web and email server so that I can turn off my main PC to save power when not in use. To store media, its had a 2TB ext4 filesystem and with the types of data, a lot of duplication. At work, I use Windows 2012 R2 and Server 2016 and often make use of the data deduplication feature which performs a scheduled offline deduplication. Offline Dedup has the benefit of not requiring much RAM but is best for static data. This is because if the volume changes alot, the dedup process can end up never finishing in maintenance windows.
But, I digress. On Linux there are a few solutions, but ZFS dedup is online and requires extraordinary amounts of RAM. BTRFS is fast becoming a favorite filesystem and replacing ext4 and it is far more feature rich.
The Raspbian kernal has btrfs support out of the box. You can check this on any Linux system by typing : cat /proc/filesystems | grep btrfs btrfs
The conversion
At first I read up on the documentation of a btrfs-progs utility called btrfs-convert. This tool is supposed to safely in-place convert an ext2/3/4 volume to btrfs. I first tried running this from the Raspberry Pi, but monitored the process using top and iotop. After running for 8+ hours I gave up and ctrl-C'd the process. Luckily this is safe to do and the ext4 metadata is still intact.
I then plugged the USB 3 volume into my big honking Fedora box and ran the con- version. It was much quicker here but it errored out with libc errors. Forum searches on this issue reveal that this has apparently not worked on recent ker- nels.
Conversion failed. Backup and Restore
Method 3, I now scraped together enough storage elsewhere to backup the data , reformat as btrfs and then reload the data. Here is the command I used to format: mkfs.btrfs /dev/sdd
Tags: btrfs, raspberry-pi, linux, filesystems
Raspberry Pi Hosted Site!
I remember the old days of the internet when creating your own site was fun.
You had get a domain, sort out some DNS hosting, setup a mail server MX records, apache html. There was some real meat to the task.
Those days are over for most people as free hosting on wordpress sites and blogging engines are so easy, the old way is not worth the time.
However, some things about the internet are lost to that era. Remeber navigating a site without 6 bajillion ads trying to get in your face? Remember when a page load was only a couple of hundred kilobytes at most? In 2014 gigaom reported the average website was around 2mb and I'm sure some of the main offenders are way off the charts.
With all this in mind I wanted to bring my blogging back to a simpler time and see if I could have a nice site, hosted on the cheap and stuff.
Here are some of the attributes of the site:
- It's all hosted on a single Raspberry Pi 2
- Uses static web pages generated manually using a bash script
- Writen in plain text markdown converted using a perl script to html.
- Hosted using the light on resources ngnix web server
- Encased in an old codrel pillbox
- Sitting under the router in the kitchen!
Tags: raspberry-pi, linux, markdown