Clean Up File Names

Cleaning up Funny File Names
Keep your Original files somewhere SAFE as a source to restart the project should it hiccup : )
Utilities Used: Google them for your Platform…
Examples are for Image Files- Suit Yourself here though.
Red is Commands- Green is My Results.

detox, exiftool, imagemagick, convmv, 

I copied a lot of cranky old 20 year old Floppy Disk image files into a Linux folder to clean up with the intention they should end up inside Apple Photos which would use their proper Image Timestamp to good effect : )

$ls -l
-rwxr-xr-x 1 sysop sysop 59993 Mar 29 2003 <A9>2002 12 19 Sunrise -5<B0> (21).jpg
-rwxr-xr-x 1 sysop sysop 78345 Mar 29 2003 <A9>2002 12 19 Sunrise -5<B0> (79).jpg
-rwxr-xr-x 1 sysop sysop 55210 Mar 29 2003 <A9>2002 12 31 Silvester (1).jpg
-rw-r--r-- 1 sysop sysop 55302 Mar 29 2003 ©2002 12 31 Silvester (2).jpg
-rwxr-xr-x 1 sysop sysop 190714 Feb 15 2003 20%20Mutterstuten%20mit%20Fohlen.jpg

Be Nice! Lets Set ’em all to reasonable Permissions:
$chmod -Rvc 644 *

This untangles ‘funny’ characters and irregularities:
$detox -r -v *

These three unify foreign language characters to standard UTF-8
(Note the final "." period meaning "Here")
$convmv -r -f windows-1252 -t UTF-8 .
$convmv -r -f ISO-8859-1 -t UTF-8 .
$convmv -r -f cp-850 -t UTF-8 .

Clean Em Up!! Lowercase names:
$for file in $(ls); do mv -i ${file} ${file,,}; done
Replace spaces in file names with underbar:
$rename ‘s/\s/_/g’ ./*.jpg

None of this so far changes the original Time Stamp on the file-
Its Creation date, only its access point: Its Name.
Which is Good. Could be useful.
Older digital Pictures did not use the EXIF metadata
that records when the Picture was taken, etc, so, this is all we have got: the File Creation Date listed by “ls -l “.

I wanted to standardize on .JPG files, as there were a mix  of GIF, BMP, etc etc.. Your choice however.
ImageMagick's MOGRIFY is good for that; Here, making all gifs into jpgs.

$mogrify -format jpg *.gif

*note* I found some animated GIFs and the result was an array of single .jpg's
as jpg does NOT have the ability to Animate! Ugh!
IE: This single file has 6 images within it. using ImageMagicks' "identify" utility.. Just a Warning...
$identify WdfAnimate.gif
WdfAnimate.gif[0] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000
WdfAnimate.gif[1] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000
WdfAnimate.gif[2] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000
WdfAnimate.gif[3] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000
WdfAnimate.gif[4] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000
WdfAnimate.gif[5] GIF 275x440 275x440+0+0 8-bit sRGB 256c 98.7KB 0.000u 0:00.000

The resulting new JPG output files have today’s timestamp,
not that of the original GIF, So! ::

$for i in *.gif; do touch -r "$i" "${i%.*}.jpg"; done

This ‘touches’; (Sets the Timestamp) as the SAME as a Reference file– the Original.
Now Let’s mess with the file “witch.gif” for Testing; Then apply to All..

$jhead -exifmap witch.jpg
File name : witch.jpg
File size : 67559 bytes
File date : 2000:02:16 08:35:32
Resolution : 398 x 300
JPEG Quality : 92

No EXIF data Present. Let’s create it with the current File Date:
$jhead -mkexif witch.jpg
Modified: witch.jpg

Now Look: Additional MAP file EXIF records the EXIF
Timestamp as if when Picture was Taken
regardless of what happens to the file’s timestamp from here on out:

$jhead -exifmap witch.jpg
Map: 00008-00038: Directory
Map: 00038-00058: Data for tag 0132
Map: 00058-00076: Directory
Map: 00076-00096: Data for tag 9003
Map: 00096-00126: Directory
Map: 00126-00126: Thumbnail
Map: 00126- End of exif
Map: 00000 49 49 2a 00 08 00 00 00 02 00
.. thumbnail data, I think?? ...
Map: 00120 00 00 00 00 00 00 00 00 11 04
File name : witch.jpg
File size : 67677 bytes
File date : 2000:02:16 08:35:32
Date/Time : 2000:02:16 08:35:32
Resolution : 398 x 300
JPEG Quality : 92
______________________

Use the Manual Pages for these Utilities here for much more useful stuff : )

 

HEIF, HEIC new Image FileFormat

Out of the blue almost, Well, about 2015 in fact,
it seems, came a New Image File Format:
“High Efficiency Image format” .heif & .heic
that miraculously is a third or even a half the file size of conventional JPG and GIF formats we are so used to with comparable-or even better- image quality :
It also ‘does’ a lot more and has more capabilities

Starting with  Phone 7 with IOS 11 or newer,  and contemporary  Android devices, this is the default format– and No Wonder. The ‘high efficiency’ achieved would normally be at the cost of heat and precious battery life  in portable devices due to the heavyweight software calculations involved.
But now, this process is embedded within the Silicon Chips  your device for far higher efficiency- Hardware. Tis is also generally true of Intel CPUs of ix 8th generation or higher (Since 2015 approx) 
More Here:

https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format

A comparable format for Video appeared about the same time, HEVC: High Efficiency VIDEO coding (more about that elsewhere) suffice to say I noticed without knowing why, a 50% DROP in  Face-time Bandwidth used if both ends  had at least an iPhone 7 or above :))

So your phone’s precious Storage is spared when taking still images- Or videos- Transferring them uses less of the main limiting resource on the Internet–Bandwidth, and everyone is happy, right?

Almost.
Built in to most newer devices is the ability to ‘down convert’ if sending to older formats. I assume SOME negotiation takes place but it’s relatively easy to convert formats ‘transparently, on-the-fly’ using the Silicon Chips I just mentioned, 
They are handed the job and hand it back Done.
The processing MUST be real time- albeit with some acceptable  latency-  else Converting, Streaming, Saving would impose unacceptable delays.
Minimal Buffering/Latency  is a side effect of Powerful Silicon.

How about we Linux users? I use Linux MINT TARA, based on Ubuntu 19 something,  all nice and up to date, but with a 10 year old CPU and no special graphics card GPU to help..;
No real-time stuff for me!
And sure enough no Thumbnail Displays of  .HEIF images in its native File Manager NEMO  nor edit them in Gimp, the “Graphical Image File Manipulation Program”, despite its name
etc. 

sudo apt-get  install libheif-examples
the command : “apropos heif”

Shows you now have:

heif-convert (1) – convert HEIC/HEIF image
heif-enc (1) – convert image to HEIC/HEIF
heif-info (1) – show information on HEIC/HEIF file
heif-thumbnailer (1) – create thumbnails from HEIC/HEIF files

Fun! It works, too. I shall Leave The Examples Of Batch Processing Up To The CLI Scripters : )

 

.. To be Continued …

 

 

For command line tools, add:

Linux Server Breakin Attempts

Heads Up as I am notified my Virtual Linux Server logs have suddenly starting growing much faster than usual.
Also I got a warning that Virtual Memory was Low.

This is the image after things got fixed:
Looking through logs I see torrents of failed login attempts over the SSH (Secure Shell) and FTP port (yes I am trying hard to switch to SFTP but that’s another story) at the rate of 5 per second or more at times.
Several Issues to Note:
 I moved SSHD from default port 22 to 1066 years ago. 
That was not, I thought a ‘well known port’ unless of course someone figures it out.
Had not changed it since. 
– Server auto updates itself regularly and I scan and check it manually now & then.
and there does not appear to be a crack so much as brute force attacks perhaps combined with guesswork.

  • Hackers obviously scanned & found the (years old) ‘new’ port.1066. I since moved it again.
    – Hackers then launched a barrage of brute force attempts with various names and who knows what password on that particular port. (logins fails restricted to 3 per 600s session in /ec/ssh/sshd_config)
    – Interestingly, ‘root’ was never tried (It’s disabled anyway)
    I assume as this could trigger a default alert- But: admin, demo, test etc? Of course.
    – These attacks came from unique IP addresses all over the word. Yes, folks, mainly China and Asia. Russia did not show up per se but then why would it? : )
    Few came from the same source IP or even subnet more than once. RESPONSES:
    – I tried to Ban China in Iptables. Not so easy as it sounds and a poor solution anyway being a majority of the sources, but not all.
    Overfilling Iptables uses up kernel memory and exhausts Virtual Memory : (
    – I setup “fail2ban”, which examines pre determined  log files for fails and acts upon it to ‘ban’ the source using Iptables again.
    which is useless as each attempt was from a new IP. Oh Yes! From literally YEARS ago I suddenly recalled /etc/hosts.allow & /etc/hosts.deny which act on the initial service port connection and CAN check wildcard hostnames by name AND IP.
    So now my rules are: Deny from anywhere EXCEPT couple of my local ISPs. No-one gets in now, regardless, unless their reverse IP name matches ISPs in my area.
  • a good solution would be light on server resources lest the result be a Denial Of Service attack overwhelming the system with blocking rules. 
  • Judging by what’s happening recently  I fear a “Grey Goo Meltdown” of the Internet- I assume MOST of these attacking hosts have themselves been broken into and turned into ‘zombie bots’ attempting to propagate themselves. The ultimate purpose is to obtain a concerted powerful platform running software of the primary attacker’s choosing to launch denial of service attacks on target domains
    These services are For Hire on the Dark Web.Here is a sample log at the end of this post ,
    and I am thankful my slackadaisical  inattention was not more severely punished by the blackhats of the Internet.
    I used to use hosts.allow/deny on EVERYTHING with only minor inconvenience.Security is Interesting & entertaining much like a firework display until you get blasted… : )

Mar 1 03:44:54 s19410066 sshd[5235]: Failed password for invalid user aion from 49.235.69.80 port 55082 ssh2
Mar 1 03:44:54 s19410066 sshd[5237]: Received disconnect from 49.235.69.80: 11: Bye Bye
Mar 1 03:44:57 s19410066 sshd[5348]: Invalid user odoo from 211.193.58.173
Mar 1 03:44:57 s19410066 sshd[5349]: input_userauth_request: invalid user odoo
Mar 1 03:44:57 s19410066 sshd[5348]: pam_unix(sshd:auth): check pass; user unknown
Mar 1 03:44:57 s19410066 sshd[5348]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=211.193.58.173
Mar 1 03:44:59 s19410066 sshd[5348]: Failed password for invalid user odoo from 211.193.58.173 port 34944 ssh2
Mar 1 03:44:59 s19410066 sshd[5349]: Received disconnect from 211.193.58.173: 11: Bye Bye
Mar 1 03:45:01 s19410066 sshd[5350]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=202.79.168.244 user=root
Mar 1 03:45:02 s19410066 proftpd: pam_unix(proftpd:session): session opened for user willowsweather by (uid=0)
Mar 1 03:45:04 s19410066 sshd[5350]: Failed password for root from 202.79.168.244 port 58772 ssh2
Mar 1 03:45:04 s19410066 sshd[5352]: Received disconnect from 202.79.168.244: 11: Bye Bye
Mar 1 03:45:17 s19410066 sshd[5574]: Invalid user jose from 167.172.118.117
Mar 1 03:45:17 s19410066 sshd[5575]: input_userauth_request: invalid user jose
Mar 1 03:45:17 s19410066 sshd[5574]: pam_unix(sshd:auth): check pass; user unknown
Mar 1 03:45:17 s19410066 sshd[5574]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=167.172.118.117
Mar 1 03:45:19 s19410066 sshd[5574]: Failed password for invalid user jose from 167.172.118.117 port 58284 ssh2
Mar 1 03:45:19 s19410066 sshd[5575]: Received disconnect from 167.172.118.117: 11: Bye Bye
Mar 1 03:45:20 s19410066 sshd[5576]: Invalid user admin from 139.59.13.223
Mar 1 03:45:20 s19410066 sshd[5577]: input_userauth_request: invalid user admin
Mar 1 03:45:20 s19410066 sshd[5576]: pam_unix(sshd:auth): check pass; user unknown
Mar 1 03:45:20 s19410066 sshd[5576]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.59.13.223
Mar 1 03:45:22 s19410066 sshd[5576]: Failed password for invalid user admin from 139.59.13.223 port 59480 ssh2
Mar 1 03:45:22 s19410066 sshd[5577]: Received disconnect from 139.59.13.223: 11: Bye Bye
Mar 1 03:45:44 s19410066 sshd[5584]: reverse mapping checking getaddrinfo for 187-45-103-15.mhnet.com.br [187.45.103.15] failed – POSSIBLE BREAK-IN ATTEMPT!
Mar 1 03:45:44 s19410066 sshd[5584]: Invalid user time from 187.45.103.15
Mar 1 03:45:44 s19410066 sshd[5585]: input_userauth_request: invalid user time
Mar 1 03:45:44 s19410066 sshd[5584]: pam_unix(sshd:auth): check pass; user unknown
Mar 1 03:45:44 s19410066 sshd[5584]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=187.45.103.15
Mar 1 03:45:46 s19410066 proftpd: pam_unix(proftpd:session): session opened for user artol by (uid=0)
Mar 1 03:45:47 s19410066 sshd[5584]: Failed password for invalid user time from 187.45.103.15 port 55849 ssh2
Mar 1 03:45:47 s19410066 sshd[5585]: Received disconnect from 187.45.103.15: 11: Bye Bye
Mar 1 03:45:58 s19410066 sshd[5589]: Invalid user demo from 49.234.60.13
Mar 1 03:45:58 s19410066 sshd[5590]: input_userauth_request: invalid user demo