MPAPI Forum Rss Feed

New Post: FYI: Builds/Compiles Fine Under .NET 4.0

ChuckMoore — Fri, 31 May 2013 16:43:30 GMT

Hi,

Just to let y'all know: built/compiled it under VisualStudio 2010 using .NET 4.0 (SP1). Everything works fine.

Am extending it to execute PowerShell (PowerShell script sets-up disk-files, cleans scratch/output areas, eventually calls C#/F#/IronPython executable which does heavy-lifting, PowerShell then does clean-up/filing-of-results).

Chuck

New Post: Registration Server Redundancy

ffrenzel — Mon, 09 Apr 2012 19:38:19 GMT

I apologize in advance, I am new to distributed computing and want to impart sufficient detail about the design of our system to make sure that

Background & Design:

We are an organization with a large number of system, build around a core OLTP system (which I will refer to as our Case Manage System), that works with entities we will refer to as Cases.

The case management system supports case life cycle management across a number of products for multiple clients, and is relatively extensible. Its biggest short coming is that it has a number of legacy legacy clients, but no central service tier (clients include intranet portal for internal employees, internet portal for external users, and tablet devices for both internal and external users).

Our business needs with in each product managed in the case management system vary greatly, and we have a number of ancillary sub-systems to extend the core case management capabilities for additional needs (document generation (letters & fax), report generation, real time export to client systems, telephony integration).

As we approach a systemic refactor of the clients participating in this case management platform, I would like to start building the central service tier we currently lack. The general approach we would like to take is to build a cluster of service hosts (collectively known as the Case Action Controller) that would receive a message from any of the case management clients at predefined events in the case life cycle, and forward that message to services that have subscribed with the Case Action Controller to receive specific messages (each of these known as Case Action Listeners).

Case Management clients will always address the Case Action Controller from a single load balanced address, the node in the cluster that receives the message will forward the meesage to any registered Case Action Listeners who've expressed an interest in cases of that type (the criteria for this determination aren't pertinent right now).

As our business works, I can afford a single point of failure at the Case Action Listener level (I can requeue most of this manually at a later time), but not with in the Case Manage System or the Case Action Controller.

I would like to use MPAPI to build this CaseActionController level (partially to handle load, partially to avoid the controller becoming a single point of failure). I would like to start up N nodes on N hosts, all listening on the same port.

Questions:

Is MPAPI a good fit for me. Part of what I am concerned with is that I actually need a cluster infrastructure that help nodes maintain a consistent state for some of my use cases (i.e. the nodes need to share the collection of registered case action listeners so that they can forward case events to all appropriate listeners), otherwise it would be a simple web farm.

I am concerned that, from what i've read, there appears to be only one Registration Server in any cluster. Is it true that a cluster has only one registration server? Is it possible to make this registration server redundant, or is there another mechanism to provide fault tolerance?

I'm sure I'll have other questions as I dig in, but this is enough while I'm getting started.

Thanks

New Post: C# Mono using MPAPI

xepharalon — Sun, 29 Jan 2012 23:21:13 GMT

I cannot compile my C# program in mono using the MPAPI. The error I get is:

Program.cs(5,7): error CS0246: The type or namespace name `MPAPI' could not be found.
    Are you missing a using directive or an assembly reference?
    Compilation failed: 1 error(s), 0 warnings

Running Mono 2.0...how do I correctly include MPAPI in my compile or program so that it builds successfully?

New Post: .net 4

kelus — Sat, 05 Nov 2011 21:26:47 GMT

Hello, is this project works properly when was it build in .net 4?

New Post: Calling methods on the MasterWorker

blbrodie — Mon, 10 Oct 2011 15:38:22 GMT

Please correct me if I missed something. Also, I am using this on Mono.

The examples given don't reflect how a master worker would likely be used within a context of a larger program. The problem is, there is no way to set parameters or call methods on the MasterWorker. This makes the framework fairly useless. Consider using this framework as a web service... how could this be done with the current design?

OpenDistributed<TRootWorker(...) returns the specified worker, but when I called methods on it from within Program I would get the error "Node.PutMessage : Cannot dispatch message"

Secondly, as far as I can tell, there is no need to launch the MasterWorker in a separate thread. This is overkill, as generally one would want to run the MasterWorker in the main thread. If not, one could always run it in a different thread manually.

I was never able to diagnose the exact cause of the error, but I was able to make a new public method within the Node class which fixes the problem. This method returns the worker and adds the node to the worker. One can then call the methods from the main thread (such as Main() or DoThisOperation() or RunWithParamaters(paramaters)) etc...

		public TMainWorker SetMainWorker<TMainWorker>(string registrationServerNameOrAddress, int registrationServerPort, int listenerPort)
			where TMainWorker : Worker
		{
			OpenAndConnectToRegistrationServer(registrationServerNameOrAddress, registrationServerPort, listenerPort);
			
			    ushort workerId = 0;
                while (_localWorkers.ContainsKey(workerId))
                {
                    if (workerId == ushort.MaxValue)
                    {
                        Log.Error("Unable to spawn more workers. There are already {0} workers running", _localWorkers.Count);
                        return null;
                    }
                    workerId++;
                }
			
				TMainWorker worker = Activator.CreateInstance<TMainWorker>();
	   
			    worker.SetNode(this);
                worker.Id = workerId;
                
                _localWorkers.Add(workerId, worker);

                if (_localWorkers.Count >= WorkerCountWarningLimit)
                    Log.Warning("There are now {0} workers on this node.", _localWorkers.Count);

			
			return worker;
			
		}

Just add that method to Node class and call a MasterWorker as follows:

worker = node.SetMainWorker<MainWorker>(regServerAddress, regServerPort, port);

worker.RunStuff();

Please advise if I misunderstood something. Or have anything to add.

New Post: log4net

azsaavila — Fri, 19 Dec 2008 20:10:55 GMT

Hello there:

I'm planning to use this framework on my app, but I need log4net because that was I used all over my app. I will modify the sources to let me choose what log type I want to use. Then I will send you the changes. Can I make that, or there will be problems with the license of the MPAPI.

Thanks.

New Post: DistributedPrimeCalculatorApp

Caduceuscoil — Wed, 30 Jul 2008 18:36:47 GMT

Frank,

That's why it's an opensource project, others can respond and help it along as well :)

New Post: DistributedPrimeCalculatorApp

fthomsen — Wed, 30 Jul 2008 08:11:43 GMT

MPAPI v. 1.1.2, which fixes the above mentioned problem, has now been released.

I am sorry for the inconveniences this has brought.

/Frank

New Post: DistributedPrimeCalculatorApp

fthomsen — Wed, 30 Jul 2008 07:23:07 GMT

Apparently I was too busy with the last release and had been testing up against an older version of the .dll without noticing it. This is rather embarassing and I do hope that you can overcome this problem. I will fix it as soon as possible myself, as well as revise my testing procedures.

/Frank

New Post: DistributedPrimeCalculatorApp

Caduceuscoil — Tue, 29 Jul 2008 22:48:55 GMT

MrCoder,

I encountered the same problem, and I traced it down to the WorkerAddress class is not marked as serializable.

Put a [Serializable] attribute on the WorkerAddress class and recompile the sample, and that should fix you up.

[

Serializable]
public sealed class WorkerAddress

C.

New Post: HPC for workstations is wide open opportunity

mrcoder — Mon, 28 Jul 2008 22:18:42 GMT

I didn't mean to make anyone feel bad that they can't spend time on something or other; rather, my intention was to make people feel good by enjoying the possibilities that an opportunity might be presenting. The glass is half-full my friend!

New Post: DistributedPrimeCalculatorApp

mrcoder — Mon, 28 Jul 2008 22:06:19 GMT

The DistributedPrimeCalculatorApp does not compute, and it logs an error:

"Remote node 0 appears to be offline"

in the master's console when a 2-node cluster (1 master, 1 worker) is launched on a single-CPU workstation, in the order

Registration server --> worker node --> master node

Both nodes are reported as being registered OK in the Registration server console however. I believe the configured IP addresses and ports are OK.

The MPAPI and RemotingLite versions being used are the most current one as of July 28.

Is there a confirmation on this same behavior by anyone else?

Thanks for reading.

Geoffrey

New Post: HPC for workstations is wide open opportunity

fthomsen — Sun, 27 Jul 2008 10:29:42 GMT

I most certainly agree with you that there is a great oppotunity here to fill out the gap on workstations. Message passing has long been regarded as either esoteric, not worth the trouble, or something for massive parallel systems (like supercomputers). But as you say the number of cores on workstations will increase, and furthermore there are a number of systems that must run on a (compared to a supercomputer) small number of nodes. And the developer community is finally starting to realize that their software is not scalable, and that multithreaded programming should be employed more to utilize the new multicore processors. This is where frameworks like MPAPI, and programming languages like Erlang and F#, fits like a glove.

Your idea of making a standardized interface for these small-scale parallel frameworks is great. However, I don't think anyone could agree on such an interface (or programming model, whichever fits). This would demand that the different projects work more closely together - something I can't see happening right now. But hopefully my pessimism is not warranted.

I am well aware that MPAPI does not have features like collective operations or cluster partitioning, but the intention was to create a framework which did not have a steep and long learning curve to figure out. But that is not to say that these features should not be implemented in the future.

The problem with MPAPI right now is that I am the only one doing work in it - in my free time. That is not a compliant, by the way, but just a statement that MPAPI is not the only thing I am working on. I would love to give it more attention, but right now (for the next couple of months that is) it is not possible. A larger user base would certainly also induce me to do more work on it. Not that I am not grateful to those who already use it - not at all! Even a single user would make it worth the trouble, but such frameworks are complicated and hard work is needed.

New Post: HPC for workstations is wide open opportunity

mrcoder — Wed, 23 Jul 2008 15:21:51 GMT

MPAPI and some other "small" MPI-ish message passing implementations work fine on multi-way workstations as well as multi-way servers.

I think it is interesting to discuss that Microsoft's MS-MPI is being designed to work on servers specifically Micrsoft Server 2008. However Microsoft for their own reasons are not attempting to support MS-MPI on the workstation.

Well, workstations only have, what, 4 CPUs you say? Sure, but there are a number of workstation applications that can benefit from the local parallel programming model. There is no going back, workstations are going to have 2, 4, 8, more CPUs and this is the case today already. And as well, applications could also participate in distributed parallel operations with some amount classical cluster servers.

There is an opportunity now due to Microsoft abandoning message passing on the workstation, to make "the best" MPI for the Windows workstations.

MPAPI might fit that need well, or maybe some adjustments could be made to improve this opportunity. There are now several "small" message passing frameworks which are suitable for maximizing workstations performance, MPAPI being one of them. MPI.NET, PureMPI.NET are possible alternatives.

Which MPI framework will take up the workstation opportunity that Microsoft has chosen to abandon?

MPAPI does not have collective operations, but maybe this is perfectly OK on a workstation which won't have hundreds or thousands of nodes. You can just program your message passing using singular operations since there are only a handful of processors on a local computer or a small network.

Message passing suits massively parallel applications, but message passing also supports "slightly parallel" applications also. And there are some of these applications.

I think there are going to be a number of applications designed to work on workstations that could benefit from message passing parallelism.

Real time camera feeds for example is a situation that an application I am developing could definitely benefit from a dedicated processor for receiving the camera's data. If I had two cameras my single-CPU workstation would be unable to keep up. And add the load of analyzing the photos in real time, and saving the interesting photos to disk devices, and displaying the photos on the GUI, I definitely have a need for multiple processors on my workstation and message passing is key to making this work in my system. Microsoft is unwilling to let people use MS-MPI on non-servers so I have to use something else on my workstation to develop parallel code which is based on the message passing paradigm. MPAPI is filling that bill today.

MS-MPI and MPAPI have different API designs so I am in a bit of trouble in my example application, if I choose for some reason to integrate the two systems.

It might be very nice to have a smooth or transparent message passing API for workstations and servers when you want to scale. It's too complicated and messy to use 2 different message passing APIs in the same application.

Does anyone agree with me that there is an opportunity here for supporting multi-way workstation developers with a message passing framework to fill in the hole while Microsoft has chosen to forfeit the workstation message passing and instead, chase increased sales of Microsoft Server 2008?

New Post: Integration with Windows Forms

fthomsen — Tue, 15 Jul 2008 11:08:16 GMT

Hello

This will be s short post without any concrete code since I am out of the country on vacation and I only have net access for a short period.

But what you wrote is one of the problems with this "disconnected" approach - there is no nice and clean way to couple the "ordinary" world (i.e. WinForms) with the MPAPI framework. What I have been thinking about is this (just an idea - I haven't tried it yet):

The worker that needs to communicate some kind of state to the winforms part of the application will have to expose one or more events. When this event is fired, the winforms part of the app will look inside the worker for the state. Remember to lock vital parts of the worker with a monitor or the lock-statement!
This should enable the winform controller to get state from the worker (or workers). Just remember that you cannot update a control from another thread. Use the Control.Invoke method on your form (for example)

I hope this clarifies the problem a bit. Otherwise please write to me again and I will address it as soon as possible.

Best regards.
Frank

New Post: Integration with Windows Forms

SuperJMN — Sun, 13 Jul 2008 15:32:06 GMT

Hello! and thanks for your great contribution. I'm really interested in your project since I'm learning concurrent programming and I always wanted to program elegantly. And your solution is really elegant :)

I used you MPAPI yesterday to simulate an airport, with the control tower and the airplanes. It was a console application and it went OK. But now my goal is to develop a small application that is able interact with Windows Forms. I thought of a Form that loads lots of images in order to add them to a ImageList as thumbnails. The image decoding of each picture is a lenghtly process and I want the user would be able to scroll throught the ListView that contains the thumbnails being added.

I think it would be really useful for tasks like this one, but I don't figure out how to do it and how to communicate a Node with the main thread. Please, give me(us) some example.

Your help will be appreciated!

Regards.

New Post: have some problems and questions

mrcoder — Fri, 11 Jul 2008 20:04:17 GMT

Just to be clear: It is really true, I am not joking: The order in which I launched the programs originally, and the first several attempts, was followed precisely according to the instructions: 1-RegistrationServer, 2-Slave, 3-Master. I can understand why the data in the screenshots I supplied may confuse some readers. Do not read too much into the screenshots. One or more of the screenshots may be from a different later run, in which I tried the other possible orderings of master and slave for completeness after the recommended ordering failed several times (reg, sl, mas).

New Post: have some problems and questions

mrcoder — Fri, 11 Jul 2008 19:51:55 GMT

fthomsen wrote:
ad 4:
I may have been vague in the documentation regarding the sequence that the applications must be started up in. That will be changed in the next release (although at the moment I have nothing planned). The sequence is this:
1) RegistrationServer.exe
2) DistributedPrimeCalculatorApp.exe. Choose "slave" mode.
3) DistributedPrimeCalculatorApp.exe. Choose "master" mode.
That should do the trick.

If you have any other problems, don't hesitate to write me.

Hi,

I am getting this "node 0 offline" error in the demo app. What could be the problem? It's repeatable on both of the Windows XP computers that I have access to.   A single computer is where all software is being attempted. Port 8000 is being used by my registration server, port 8001 is being used by my slave node, and port 8002 is desired for my master node. I think its a good policy to get the demos working before troubleshooting my own apps.

C:\Program Files\MPAPI\MPAPI_1.1.1_example\bin>DistributedPrimeCalculatorApp.exe
[m]aster or [s]lave > m
This nodes port number > 8002
15:19:58:565 | Info      | Main worker online
15:19:58:575 | Info      | Spawning 1 workers at node 0, address 192.168.1.100
15:19:58:615 | Error     | Node._Monitor(0@1, 0@0) : Remote node 0 appears to be offline

The above master node, is the 2nd node that was started. There is already a slave node running (not shown).

In another command shell, the registration server displayed some info which seems OK.

C:\Program Files\MPAPI\MPAPI_1.1.1_src\bin>RegistrationServer.exe
15:18:29:998 | Info      | Registration server is running.
15:18:41:444 | Info      | Registered node. Node Id : 0 , Address : 192.168.1.100 , Port : 8002
15:18:54:042 | Info      | Registered node. Node Id : 1 , Address : 192.168.1.100 , Port : 8001

The order in which I launched these programs was per the instructions: 1-RegistrationServer, 2-Slave, 3-Master.

Any suggestions to fix it? (Again, this is 100% local execution, strictly one computer is running all demo software at this point.)

Thanks for reading.

Geoffrey

BTW the RemotingLite demos are working good now on both of my Windows XPSP2 computers. Uninstalling IPv6 was what solved the problem on one of these computers. To uninstall IPv6, unchecking the checkbox was not enough; I had to also click an uninstall button (which I did not notice before).

New Post: Problem: Could not connect to registration server. Message reads : An address incompatible with the requested protocol was used

mrcoder — Fri, 11 Jul 2008 18:06:19 GMT

On my friend's XP Workstation the example programs for RemotingLite both work perfectly. Your code is good Frank.

I guess that means my own XP installation on my own computer is garbage perhaps.

My friend's XP has strictly IPv4 installed, as confirmed by Control Panel | Network | Local Area Network Connection 1 | TCP/IP | Properties, and also by IPCONFIG.EXE executed in a command shell. There is no sign of any IPv6 capability on this computer. I wish I could get mine to be this way too.

I really dont know how IPv6 got onto my computer because I certainly dislike what it does to .NET 2.0 in my experience!!

I would like to know if other people's experience also indicates problems, or not, with IPv6 and .NET 2.0 both on the same Windows XP box.

New Post: Problem: Could not connect to registration server. Message reads : An address incompatible with the requested protocol was used

mrcoder — Fri, 11 Jul 2008 17:36:57 GMT

Hi,

I still have the same basic connectivity errors on my Windows XP host unfortunately after the update to latest versions. Maybe my OS needs to be wiped off and reinstalled? uggg...

Anyway, here is what displays when I use the new versions of your software. I have carefully removed the old version of all the MPAPI software and dependencies into a hidden directory to prevent confusion about versions with the new one. I have freshly recompiled all your code from source.

This is RemotingLiteExampleServer:

C:\Program Files\MPAPI\RemotingLite_1.2.4_example\RemotingLiteExampleServer\bin\Debug>RemotingLiteExampleServer.exe
Host is running. Press <ENTER> to terminate.
Address is ::1
Press <ENTER> to terminate.

That's a funny address. I think it's IPv6 unfortunately. Vista is not my OS.

What address do other people see displayed when you run this program on XP?

And here is what displays in the console when I run the client example program:

C:\Program Files\MPAPI\RemotingLite_1.2.4_example\RemotingLiteExampleClient\bin\Debug>RemotingLiteExampleClient.exe
Using our own implementation that inherits from ClientBase<>

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocati
on. ---> System.Net.Sockets.SocketException: An address incompatible with the requested protocol was used
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.Net.Sockets.TcpClient.Connect(IPEndPoint remoteEP)
   at RemotingLite.Channel..ctor(IPEndPoint endpoint)
   at IServiceProxy..ctor(IPEndPoint )
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle._InvokeConstructor(Object[] args, SignatureStruct& signature, IntPtr declaringType)
   at System.RuntimeMethodHandle.InvokeConstructor(Object[] args, SignatureStruct signature, RuntimeTypeHandle declaring
Type)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, Cultu
reInfo culture)
   at RemotingLite.ProxyFactory.CreateProxy[TInterface](IPEndPoint endpoint)
   at RemotingLite.ClientBase`1..ctor(IPEndPoint endpoint)
   at RemotingLiteExampleClient.ClientProxyImpl..ctor(IPEndPoint endpoint) in C:\Program Files\MPAPI\RemotingLite_1.2.4_
example\RemotingLiteExampleClient\ClientProxyImpl.cs:line 26
   at RemotingLiteExampleClient.ExampleUsingClientProxyImpl.Start() in C:\Program Files\MPAPI\RemotingLite_1.2.4_example
\RemotingLiteExampleClient\ExampleUsingClientProxyImpl.cs:line 23
   at RemotingLiteExampleClient.Program.Main(String[] args) in C:\Program Files\MPAPI\RemotingLite_1.2.4_example\Remotin
gLiteExampleClient\Program.cs:line 13

C:\Program Files\MPAPI\RemotingLite_1.2.4_example\RemotingLiteExampleClient\bin\Debug>

The "incompatible address" error is still breaking my RemotingLite and therefore my MPAPI installation.

I won't even bother to try the demonstration programs for MPAPI since they depend on RemotingLite working correctly.

It would be a big waste of time to erase my system disk and install a whole new OS. Therefore I hope to find a way to fix my OS installation more quickly if that's really the problem here. It is starting to seem that maybe it's my OS because since nobody else is reporting these problems with RemotingLite on XP.

With luck, I might find that there is way to completely disable IPv6 from WindowsXP.

I am going to borrow someone else's XP computer. Today I found someone who can let me try RemotingLite on their computer.

If RemotingLite demos run OK on their computer then I guess I will know it's time to discard my XP Workstation installation into a garbage dumpster. I have had problems in the past with COM services on my computer, so my working theory is that my COM services are partially broken, and remoting is broken too as a result. It's a shame because this computer's OS installation is only 1 year old. I've never had a Windows installation or registry go bad so quickly before.

Thanks for reading.

Geoffrey