Pratik Sarkar writes:

Coming back to the project,do we need to write patches for the SpamAssassin or SpamBayes(which are integrated in Mailman)

No. You write a Handler which delegates to those external packages. A Handler is a sort of plug-in.

or do we need to start over and make a whole new filter and start reinventing the wheels?

Don't do that unless you think you can write a filter that does a significantly better job on some dimension than the existing ones.

And somewhere I read that the filtering of messages posted by non-subscribed members are done manually.

That's incorrect. Mailman's mail processing is architected as a pipeline of Handlers. Each Handler can act as a filter, or edit the message, or perform a side effect (delivering the mail to subscribers or saving it in an archive), then passes the message on to the next Handler (unless it raises an exception). Filters have a selection of options: throw away the message silently (DISCARD), return to sender (REJECT), quarantine (HOLD), or pass on for further processing. (The words in all uppercase are actually exception names.) One of the filters checks whether the mail is from a subscriber, but it is not otherwise special, and in principle both subscribers and non-subscribers may be subject to various filters.