nemozone

a zone for no one and everyone :) Btw this blog is only for adults! Dieser Blog ist nur für Erwachsene!

“ezFIO – Powerful, Simple NVMe SSD Benchmark Tool

Article by Earle F. Philhower III, Western Digital Senior Manager, SSD Technical Marketing

Organizations’ users are adopting NVM Express attached enterprise SSD (eSSD) products at phenomenal rates. Increasing data volume and velocity require hundreds of thousands of IOPS and the gigabytes per second of bandwidth these new devices can provide. Today’s applications have higher parallelism and sustained performance needs, coupled with a need for predictable, consistent performance even when a device has been fully written and overwritten.

There are many benchmarks for peak metrics like IOPS and throughput. However, few address the non-stop nature of enterprise storage needs. So customers are left to develop their own methods. But writing a repeatable, meaningful benchmark is hard. Even more difficult is the task of presenting meaningful results in a spreadsheet or graph without the nightmare of manual cut-and-paste efforts.

That’s where ezFIO comes in: https://github.com/earlephilhower/ezfio . ezFIO is a Linux and Windows wrapper for the cross-platform FIO IO testing tool. It always includes a repeatable preconditioning (also known as “seasoning”) stage, to help simulate the true long-term performance of eSSDs. A wide variation in IO sizes and parallelism is run automatically.

ezFIO also includes long-term performance stability measures, which allow for latency outliers and deviations to be numerically represented and graphically identified quickly. These new SSD metrics show Quality of Service, both at a Macro level (standard deviation over time) and Micro Level (measurement of minimum and maximum latency outliers). And, finally, ezFIO takes all these results and produces an Open Document formatted spreadsheet usable under Linux or Windows with embedded graphs and raw test results to make examining these results consistently derived and for any NVMe device to be compared.

fig_1

Figure 1 — Select Drive Under FIO

To run ezFIO under Windows, simply install FIO and run a PowerShell script. Then select a drive, hit OK, and come back in several hours to an ODT formatted spreadsheet. ezFIO takes care of preconditioning the drive, running repeatable tests, and plotting the results. There is also a command-line option to allow for scripted runs against multiple devices in sequence, without user intervention. The Linux version is written in Python and implements the exact same sequence of preconditioning and tests, taking its drive under test from a command line parameter.

ezfio-fig-2

Figure 2 – Real-Time Status

While tests are running, intermediate results are shown (full test results are always stored in a new test directory). These results appear in a GUI dialog under Windows, and the terminal under Linux.

Finally, when the tests are completed, all the individual runs are collated into a single spreadsheet in the ODT format (accessible under Microsoft Excel, Open Office, Libre Office, or others):

fig3-a

Average IOPS: 293,821

Standard Deviation: 8,363.7 IOPS

Coefficient of Variation: 3%

Figure 3 — Macro QoS (Stability Over Time)

ezfio-fig-4-1024x514

Figure 4 – Pre-generated Graphs for Immediate Export

A full user’s guide is available in the package, and since the software is completely open sourced and written in scripting languages (PowerShell for Windows, Python for Linux), it’s easily modifiable for power users without needing a complete development toolchain.

ezFIO represents the next-level of performance benchmarking for NVMe, taking into consideration many critical parameters that will yield the best possible results for most use cases.

Give ezFIO a try at https://github.com/earlephilhower/ezfio."

Source: https://nvmexpress.org/ezfio-powerful-simple-nvme-ssd-benchmark-tool/

I'm sorry the articel is awesome but the default layout and font gives me eye cancer

Vergleich.org versteht sich als Service für den Verbraucher: Wir möchten Kaufentscheidungen für Konsumenten vereinfachen. Unsere Stärke sehen wir darin, Daten zu strukturieren und verbraucherfreundlich aufzubereiten. Unsere Redakteure nehmen sich die nötige Zeit, um sich intensiv in Themengebiete einzuarbeiten und die wirklich relevanten Informationen für Verbraucher zu ermitteln.

https://www.vergleich.org/ueber-uns/

Ein Beispiel für einen SD-Kartenvergleich

https://www.vergleich.org/sandisk-micro-sd/

Eliminate the barrier between your machines. Find releases for windows and macOS here. Your distro probably already has barrier packaged for it, see distro specific packages below for a list. Alternatively, we also provide a flatpak and a snap[…]

https://github.com/debauchee/barrier

Während ich als Support-Äffchen tätig war. Hatte ich immer wieder das Vergnügen, mit unterschiedlichsten Datenblock-Geräten zu arbeiten.

Ein besonderes Plä­sier hierbei stellen fehlerhafte USB-Speicher dar.

Selbst heutzutage kommt es immer wieder vor, dass ich damit konfrontiert werde.

Das Leben ist zu kurz und schmerzhaft, um es sich mit solchen dem Delir entsprungenen, aberwitzigen Müll zu vergeuden.

Deshalb habe ich immer nach Wegen gesucht, um möglichst schnell und einfach so etwas auszuschließen.

Die Bilder oben zeigen echte und nicht fehlerhafte Sticks. Die Bilder unten exemplarisch fehlerhafte oder nicht echte Sticks.

Sticks which are aight

Sticks which ain't aight

Quelle für die Bilder https://www.ventoy.net/en/doc_fake_usb.html

Zitat von der Ventoy Seite, übersetzt von mir.

Gleichwohl der spezifische Herstellername nicht wie oben in den roten Kästchen angezeigt wird, müssen Sie vorsichtig sein. Es kann gefälscht oder fehlerhaft sein. Ich habe viele Berichte über gefälschte USB-Sticks wie diesen erhalten, insbesondere solche mit dem Namen Generic xxx.[…]

Darüber hinaus werden auch einige Werkzeuge empfohlen, mittels derer wir erkennen können, ob ein Stick fehlerhaft oder gefälscht ist.

ChipGenius von hit00 https://www.usbdev.ru/files/chipgenius/ URWTEST https://www.usbdev.ru/files/urwtest/ H2testw https://www.heise.de/download/product/h2testw-50539 H2testw kann auch mittels Wine unter Linux verwendet werden.

Unter Linux oder MacOS können wir folgende Werkzeuge verwenden.

f3 – Fight Flash Fraud https://fight-flash-fraud.readthedocs.io/en/stable/#f3-fight-flash-fraud https://fight-flash-fraud.readthedocs.io/en/stable/introduction.html#testing-performance-with-f3read-f3write https://github.com/AltraMayor/f3 In Apfel Systemen kann dies mittels brew install f3 installiert werden, sofern Brew(siehe brew.sh) installiert ist

https://formulae.brew.sh/formula/f3#default

Falls Ihr Stick echt ist und dennoch Fehler verursacht, kann auch ein “flaky flash” vorliegen.

Zitat von der flakyflash Seite, übersetzt von mir.

Flakyflash

Wenn Ihr Flash-Stick echt ist (oder Sie f3fix ausgeführt haben, um ihn zu “reparieren”), aber Sie immer noch sporadische Datenkorruption erhalten, dann haben Sie möglicherweise einen “flaky flash”. Wenn Ihr Flash mit dem FAT-Dateisystem formatiert ist, können Sie Flakyflash verwenden, um die Flaky Datencluster zu finden und sie als fehlerhaft zu markieren. Dies kann es Ihnen ermöglichen, Ihren flash-speicher besser zu verwenden, aber Sie sollten ihn trotzdem als fehlerhaft betrachten und so schnell wie möglich ersetzen[…]

https://github.com/whitslack/flakyflash

Ein sehr guter Artikel, viel besser als meiner :D

https://linuxundich.de/gnu-linux/usb-sticks-und-sd-karten-auf-fehler-pruefen/

Data degradation is the gradual corruption of computer data due to an accumulation of non-critical failures in a data storage device. The phenomenon is also known as data decay, data rot or bit rot[…]

Source https://wiki.604kph.xyz/wiki/Data_degradation?lang=en

On Linux, using a btrfs partition plus a raid 1 for example with Borg backup is a nice, economical way to prevent this.

Timeshift also works with btrfs

Quote

Data rot cannot be avoided through incremental backups because once it happens you can only backup the rotten copy. RAID can protect you. But regular users rarely use RAID.

Few of the most advanced filesystems do take care of this using integrity checking and self-repairing algorithms – btrfs and ReFS. Btrfs is from Linux and can be used on desktops as a general purpose filesystem. ReFS, on the other hand, is applicable to Windows Server platforms. A older filesystem (still in use) with such capabilities is ZFS.

Source http://tuxdiary.com/2014/05/24/data-rot-and-how-to-prevent-it/

Further reading

http://tuxdiary.com/2014/05/24/data-rot-and-how-to-prevent-it/

And never ever forget that raids are not backups after you've implemented your raid 1 or whatever with btrfs to prevent data rot.

This rule still applies Quote

[…]the 3-2-1 rule is a good practice. 3 copies of your data, 2 local and 1 offsite.

Further reading

https://www.raidisnotabackup.com/

Further reading regarding raid

https://raid.wiki.kernel.org/index.php/Linux_Raid

In my life as a support monkey, I've come around a lot of block devices and also USB sticks. It is always a great pleasure to work with a piece of hardware which has hardware issues, not.

And I've always tried to find a way how to determine if the particular piece I'm working with is ok or not. Due to the nature of immense production of this particular hardware. And the attitude of the manufactures like we don't give an f… The levels of frustration are too big to waste my valuable life time with this cr*p

Thx to the ventoy folks, they've documented sweet ways to determine if a stick has an issue or not

https://www.ventoy.net/en/doc_fake_usb.html

Sticks which are aight

Sticks which ain't aight

Quote from the Ventoy page

However, if the specific manufacturer name is not displayed in the red boxes like the above, you must be careful. It may be fake or faulty. I have received many reports about fake USB sticks like this especially thoes with name Generic xxx.[…]

This jolly folks have also included nice references for programs which can determine if a stick is faulty or fraudulent

ChipGenius by hit00 https://www.usbdev.ru/files/chipgenius/ URWTEST https://www.usbdev.ru/files/urwtest/ H2testw https://www.heise.de/download/product/h2testw-50539 The opportunity of using H2testw via Wine or similar tools is possible :)

f3 – Fight Flash Fraud https://fight-flash-fraud.readthedocs.io/en/stable/#f3-fight-flash-fraud

https://fight-flash-fraud.readthedocs.io/en/stable/introduction.html#testing-performance-with-f3read-f3write

https://github.com/AltraMayor/f3

In Mac Systems, one can use via brew.sh(must be preinstalled) brew install f3 from the terminal to install, f3

https://formulae.brew.sh/formula/f3#default

Quote

Flakyflash

If your flash isn't fraudulent (or you've run f3fix to “fix” it) but you're still seeing some sporadic data corruption, then you may have “flaky flash.” If your flash is formatted using the FAT file system, then you can use Flakyflash to find the flaky data clusters and mark them as bad in the FAT. This may allow you to get a little more use out of your flash, but you should still consider it as failing and replace it ASAP[…]

https://github.com/whitslack/flakyflash

Installation

sudo apt-get install testdisk

Starten

sudo testdisk

sudo testdisk /pfad/zum/image.img

Offizielle Testdisk Doku

https://www.cgsecurity.org/wiki/TestDisk_DE#Dokumentation

Ur Referenz: https://wiki.ubuntuusers.de/Datenrettung/#Geloeschte-defekte-Partitionstabelle-retten

Festplatten Problembehebung

https://wiki.ubuntuusers.de/Festplatten_Problembehebung/

Dateisystemcheck

https://wiki.ubuntuusers.de/Dateisystemcheck/#Manuelle-Pruefung

Schreiben auf deutscher Tastatur mit US-Zeichensatz

Die folgende Übersicht soll Ihnen helfen, wenn Sie vor einer deutschen Tastatur sitzen und mit dem US-englischen Zeichensatz schreiben. Sie zeigt die Tasten und Tastenkombinationen für Buchstaben und Sonderzeichen, deren Positionen von der deutschen Tastatur abweichen[…]

https://computertastatur.info/schreiben-auf-deutscher-tastatur-mit-us-zeichensatz/