Author High Performance Computing  (Read 9996 times)

0 Members and 1 Guest are viewing this topic.


Re: High Performance Computing
« Reply #31 on: April 05, 2016, 11:29:18 PM »
I know what clusters are and why some people need them... what I want to know is why you need one.
I work in the field. I use it for testing and running parallel applications. :)

If there is one thing I absolutely hate, it's lag time. Parallel and distributed computing can't be done efficiently on a single threaded machine no matter how powerful it is. It's always best to develop on a platform that's as similar as possible to the one you're targeting. I also test algorithms that are extremely compute intensive. I don't have the luxury  of fucking around waiting for a compilation to complete. When I'm on the jazz, I need a machine that can keep up to me.

Re: High Performance Computing
« Reply #32 on: April 07, 2016, 04:08:13 AM »
I work in the field. I use it for testing and running parallel applications. :)

If there is one thing I absolutely hate, it's lag time. Parallel and distributed computing can't be done efficiently on a single threaded machine no matter how powerful it is. It's always best to develop on a platform that's as similar as possible to the one you're targeting. I also test algorithms that are extremely compute intensive. I don't have the luxury  of fucking around waiting for a compilation to complete. When I'm on the jazz, I need a machine that can keep up to me.

Sorry dude, but this is a worthless post.  You are talking like a drunk neck-beard.  "I'm in the field"  "on the jazz" "machine that can keep up with me"... doing WHAT exactly?   "Test algorithms" that do what?    If you said running hurricane models, RSA encryption cracking schemes, mass facial processing, AI heuristic functions or hell, even number crunching for SETI, I would have more respect for you.   

It's clear you have an idea of what you're talking about, but it isn't clear that you are doing something worthwhile with the power supposedly at your fingertips.  The fact of the matter is that most people couldn't or wouldn't know how to harness all of it, and that you have to pretty much write specific code (or work within a platform that does) to take advantage of large parallel systems.

I had a professor in college that I used to hang out with who worked at Cray.  We'd go to lunch and kayak from school to his work.  Back in those days, that was THE name of the game in terms of super computing as I'm sure you know.  So don't just spit a bunch of bullshit at us.  Pics (or equivalent) or it isn't happening..




Re: High Performance Computing
« Reply #34 on: April 08, 2016, 03:34:30 AM »
Why stop there, string together 8 of them and really have some fun.

http://www.engadget.com/2016/04/05/nvidia-dgx-1-deep-learning-supercomputer/

speaking of AI, there was a breakthrough a couple of years ago by the u of Toronto team at the image recognition competition,  and since then many people are applying their learning method to other fields.   I dug around a few months ago looking for the exact algorithm, or even some source, and wasn't able to find it.    Have any of you guys seen anything on this?   My guess is it's in some cs journal that I can't get access to without going back to my alma mater and hitting up the engineering library.   Fucking scientific journals.. they should be free and easy to access for anyone to read.    #NeverForgetAaronSwartz

Re: High Performance Computing
« Reply #35 on: April 08, 2016, 09:47:12 AM »
Sandman, you know I was just joking, right?   I just want to know what you're doing working with the technology.

Re: High Performance Computing
« Reply #36 on: June 11, 2016, 06:59:31 PM »
Sandman, you know I was just joking, right?   I just want to know what you're doing working with the technology.

SAN storage?

Re: High Performance Computing
« Reply #37 on: June 19, 2016, 04:00:35 PM »
I understand about 20% of this thread but it's awesome.  ;D

Re: High Performance Computing
« Reply #38 on: July 07, 2016, 08:17:39 AM »
Sandman, you know I was just joking, right?   I just want to know what you're doing working with the technology.
I just figured you were probing for information in an area I'm not willing to discuss. It was wrong for me to pull a Hoagland on you though, so I'll say it straight out. As a security measure, I don't talk about how I use my equipment. That's no horseshit. Ideas are stolen every day. What I can tell you is that I process real-time data. That's vague enough to to protect my intellectual rights, but should give you an idea as to what I'm using it for. I won't get anymore specific than that. We're not the only ones that read these boards.  ;)

Re: High Performance Computing
« Reply #39 on: July 07, 2016, 09:03:53 AM »
I know what it is now.
Ruh-Roh. I've said too much. :o

Re: High Performance Computing
« Reply #40 on: July 10, 2016, 09:06:17 AM »
Ruh-Roh. I've said too much. :o
Sandman's living room. Crunching SETI data :)

Re: High Performance Computing
« Reply #41 on: July 10, 2016, 01:28:23 PM »
This thread should be used for all aspects of HPC and Clustering. If you've built your own system and are proud of it, show it off here too - please include the extra goodies, specs, and bench tests if you did them.




The first question you need to ask yourself is: Are the programs you are running compute intensive, or memory / hard-disk intensive ? If you are running highly parallel programs like GIMP, POVRAY(not parallel coded), Scientific Computational Modelling, or numbers crunching programs, keep reading.

Some of the hardware factors that determine a computer's speed of execution are cpu frequency and # of cores, memory speed and rank, interconnect speed between components, disk write (and [not so much] read) speeds, PCI-e bandwidth,  NIC bandwidth and interconnect technology (copper, fiber, infiniband, etc.)

Traditional cluster computers dependended on multiple independant systems connected through an ethernet with switches controlling packet routing. Switches are expensive, contribute to latency, and are a wiring mess. 
What if you could get rid of the interconnect wiring and the switches, juice the ethernet and keep all the boards contained in a single tower with 4 nodes for under $6,000.00 ?

Worth checking out ?

One more thing, when you're not running highly parallel computational tasks, this system also doubles as a workstation. You could build yourself a system like this through lots of research, trial and error, and writing your own code to tie everything together, or you can check out the following links.

http://www.clustermonkey.net/Interconnects/experiments-with-switchless-10gige-the-bonded-loop.html - Lots of good clustering info.
Beowulf info + links - http://fscked.org/writings/clusters/cluster-1.html

-------------------------------------------------   The Limulus Project!    -----------------------------------------------------

http://limulus.basement-supercomputing.com/

Buy one pre-built or get help designing one. 

Limulus Benchmarks

Base Model $5,495.00  here,  + Spec Sheet



-----------------------------------------------   Clustering Books   ---------------------------------------------------------------------------------

http://www.clustermonkey.net/Columns/Cluster-Newbie/


---------------------------------------------------------------------------------------------------------------------------------------------------------

Down & dirty use of old computers - Beowulf.

...And just so everyone that reads this thread has an understanding of how computers work - Don't let the magic smoke escape !

Explanation:

The core of a computer is a small trapped evil spirit (it is no coincidence that UNIX and GNU/Linux processes are called daemons!) This imp is imprisoned in your computer as punishment for something it did in the netherworld. Generally speaking, the more evil the imp is, the faster the computer is. The imps inside 1,000Mhz and slower computers are usually just big fat jerks - a.k.a. Sendas. There are also lesser ghosts and hobgoblins that inhabit the expansion cards and peripherals of your computer. These evil spirits are trapped within the hardware of your computer. Take a screwdriver, open up your computer, and take a look at the circuit boards. They are usually green, and are covered with complex patterns of thin copper lines. These are Circuit Runes, written in arcane and ancient languages that describe the magic spells that bind the imps to the chips. Be very careful with circuits. If you scratch off even the tiniest bit of the runes, the spell will be broken, and the imp will escape in a puff of magic smoke. If you coincidently happen to scratch these runes while the imp is awake and plugged to power, your fingers might burn and you would be thrown back with a shock. Well, that is the evil nature of the imps. The movie series Terminator is about what could happen if the evil of the imps in computers comes to life and take control. Also note the serial and part numbers printed on the boards. These identify exactly what sort of evil spirit it is, in case you ever have to order a replacement from the factory. :o


I'm computer illiterate and have absolutely NO idea what you are speaking of here.  Strangely enough, I'm horny as hell now.  Go figure. ???

Re: High Performance Computing
« Reply #42 on: August 21, 2016, 03:51:15 AM »
I'm computer illiterate and have absolutely NO idea what you are speaking of here.  Strangely enough, I'm horny as hell now.  Go figure. ???
:-*    ;)

Re: High Performance Computing
« Reply #43 on: August 21, 2016, 03:54:07 AM »

Re: High Performance Computing
« Reply #44 on: August 21, 2016, 04:16:16 AM »
I'm computer illiterate and have absolutely NO idea what you are speaking of here.  Strangely enough, I'm horny as hell now.  Go figure. ???
lol!

Re: High Performance Computing
« Reply #45 on: August 22, 2016, 05:33:52 AM »
good thread.

Re: High Performance Computing
« Reply #46 on: August 22, 2016, 08:13:50 AM »
Thank you MV.  :)
That means quite a bit coming from you. I'll try to keep good info coming.


Re: High Performance Computing
« Reply #48 on: August 24, 2016, 04:31:45 AM »
Engineering papers:

The International Journal of Grid and Distributed Computing. All issues starting from 2008 through present. Free for download.

http://www.sersc.org/journals/IJGDC/vol1_no1.php

Re: High Performance Computing
« Reply #49 on: August 24, 2016, 04:51:19 AM »
The pegasus diskless slave node cluster.

http://web.mst.edu/~vojtat/pegasus/home.htm

Re: High Performance Computing
« Reply #50 on: August 24, 2016, 05:16:11 AM »

Vaidas Jablonskis tutorial using rpm based systems. Should also work for debs.

https://jablonskis.org/2011/howto-to-build-a-diskless-linux-cluster/index.html

Re: High Performance Computing
« Reply #51 on: August 24, 2016, 05:26:43 AM »
Run purely in RAM for security and speed. Also some good references at the end of the article.

http://eduardo-lago.blogspot.com/2012/06/ram-only-pxe-boot-smallest-diskless.html

Re: High Performance Computing
« Reply #52 on: August 24, 2016, 05:46:31 AM »
An Operating System worth checking out, weather running as a cluster or stand alone. Bare metal security distro (Qubes.)

64-bit only.

https://www.whonix.org/wiki/Qubes

Re: High Performance Computing
« Reply #53 on: August 24, 2016, 10:10:47 AM »
The pegasus diskless slave node cluster.

http://web.mst.edu/~vojtat/pegasus/home.htm

Those 6 spinning disks in the server must be under a heavy load.

I've always wanted to build a small cluster (like 5 machines) to work on a pet project I have. Software draws art.


Re: High Performance Computing
« Reply #55 on: September 04, 2016, 10:11:14 PM »
2016 International High Performance Conference at Salt Lake City in November.

http://sc16.supercomputing.org/

Re: High Performance Computing
« Reply #56 on: September 04, 2016, 11:49:55 PM »
200,000 cores?

http://www.pcworld.com/article/3111693/hardware/open-source-25-core-chip-can-be-stringed-into-a-200000-core-computer.html
Sweeeeeeet. Baked in hypervisor support plus the cores are mesh fabric topology. Hope they work out the glitches & get this bad boy out into the wild.

The K (Kei) super computer in Japan uses a six-dimensional mesh-torus interconnect topology. Although that is an inter-cpu topology, the same applies to intra-cpu topologies of which Sparc is one (Mesh).



Topologies explained:
http://15418.courses.cs.cmu.edu/spring2013/article/30

The K Mesh-Torus:
http://www.fujitsu.com/global/about/businesspolicy/tech/k/whatis/network/

http://www.fujitsu.com/global/about/businesspolicy/tech/k/whatis/system/

BTW - Nice find zeebo.  :)

Re: High Performance Computing
« Reply #57 on: September 05, 2016, 12:16:36 AM »
Those 6 spinning disks in the server must be under a heavy load.

I've always wanted to build a small cluster (like 5 machines) to work on a pet project I have. Software draws art.
This article is for CAD, but the engineering principles are sound and should be applied to a graphics HPC also.

Don't overlook the other links on that page, there are some good ones.

Re: High Performance Computing
« Reply #58 on: September 05, 2016, 03:11:38 AM »