Friday 8.10am, and a session on SOA and student data. A subject that is very hot for us at the moment as we try and implement XML .Net based SOA to tie all our student data together and deliver the information agility that our University needs.
So far and not too technical - presentation starting with quick overview of the problem - increasing need for data and information, data in lots of silo systems, access to data and reporting via 'programmers' based in IT ie: fixed reports and ad hoc reports if you were lucky. Yup - all very familiar ....
Data warehouses and BI tools allow 'Surfacing of Data' - data in the hands of the decision makers, removing or at least reducing silos, creating flexible outputs which the end user can use so that developers can get on with developing rather than having to spend all their time generating reports ..... Ok - agree with all of that ...
BI tools are now changing the expectations of decision makers - predictive modelling, decision processing, point in time analysis (not sure what that means exactly), all being demanded. However there are problems with deciding how granular you are going to need your raw data to be, the context and definitions of your data and the data refresh cycles you need - these are all issues you need to address.
(As an aside this presentation is being delivered as a classic put up the slides and then paraphrase the slide and expand on it - which makes it nice and easy for those of us who are sitting here writing notes as the presentation happens, but maybe makes it a bit slow and repetative otherwise?)
A move from static data to "Dynamic BI" means that you have to start thinking 'processes' in a different way. Process flows and workflows become important and data rules have to be agreed and understood as they are going to be automated so they have to deliver the right thing.
It occurs to me that this is the same old story that was told in the days when BPR (Business Process Re-engineering) was all the rage. Make sure that you don't pave over the cow paths. If you just automate the processes that have evolved over time then you just end up automating inefficient processes, that meander all over the place, and that don't take you from A to B efficiently. You need to start from scratch and re-engineer the processes first (so that you cross the field in a straight line rather than by means of meandering cow paths) before you automate them.
Back to the presentaion - the demand for dashboard functionality and reports also starts rising exponentially (as we have already seen with the implementation of a simple dashboard reporting tool for some of our data)
BI requires - "next-generation architecture" ie: SOA (Service Orientated Architecture). Current architectures are limited. Yup.
Buzz phrase "Change Adaptive" used by the speaker to mean we are working in an era of rapid change - legislative, compliance, business etc and systems need to change at the same speed.
Ok I was already convinced that I need SOA, and now I'm convinced all over again. A different speaker now (hope he's not going to get too techy)
How does SOA help solve these problems? - the underlying data schema can evolve with out requiring a whole load of changes to reporting and systems integration. This means you have to put a service layer inbetween the actual databases and the workflow, integration and reporting functionaliuty you want - Yup - come on we know all this thats why we're sitting here .....
(As another aside if you're reading this and you're not familiar with SOA in an HE environment then I can thoroughly reccomend this short JISC animation
http://www.jisc.ac.uk/whatwedo/programmes/eframework/soa.aspx which describes it brilliantly and simply even for the non-techie!)
Presenter now showing a schema of a generic SOA layout ..... (watch the animation above and you'll get the idea if you don't know what this might look like)
One change that this brings in is that you need a different data structure for the service layer than for the database(s) underneath it and you need to think about it. The schemas and dictonrys for the service layer are multiple and small. They need to capture 'transformational logic' by which I assume speaker means the rules by which data is changed or processed within the service layer. Each data element (object? artifact?) needs a common service definition in the service layer that applies to all databases that hold that element. What this means is that you need to treat business logic as if it is data - ie: business rules and workflow processes are now automated and therefore have to be defined and captured as data. This I think is the essential BIG change. Administrators and managers are not used to thinking of their processes and ways of working in this discplined way. But the benefits are huge in terms of flexibility of reporting and adapting to changing process requirements.
Speaker now talking through a diagam of how service adapters (chunks of code that define transformational logic and data elements) are used -a bit over my head at this point but I think I get the gist!
Adapters can be used for different things for example they can be used to carry out a mapping of a data element across two databases and carry out a transformation of that data in some way eg: changing the status of a student from application to registration. Or it can be used to access the same data element from different systems and any associated data elements and pass them up to a reporting function within the service layer (eg: tying together timetabling information and course information for a single student from the timetabling system and the student information system). (NB: the eg: are mine not the speakers so they come with a health warning as I may have totally misunderstood!)
We are now going to get a live demo on reporting via a service layer - and at this point it all gets way too technical for me. The demo essentially seemed to be using an open source 'ETL' tool (don't know what ETL is? Extraction Translation Language??) called Talend.
The demo showed to a list of data elements that were defined for use in the Service Layer. Woa! - there's a whole load of code on the screen and most of the blokes with beards are all leaning forward and concentrating intently! I feel like an 'under-cover' interloping CIO! - 3 people have just left ...... maybe they're CIO's too ....
Think I'll stick it out but what would be helfpful for me is to hear more about the challanges of actually implementing this stuff. How long did it take, what where the obstacles, what are the limitations and the BIG question - how do you manage to get the stakeholders to sit down with you to define the processes and data elements that are needed to create a service layer?
Speaker now making the point that you can create user interfaces for different purposes to the data in the service layer which comes from different systems. Now showing some fancy open source reporting tools that you can layer on top of your service layer - much like we will do with our reporting tool Qlikview which when we get SOA in place will sit on top of the service layer instead of sitting on top of data extracted and combined into spreadsheets as at present.
Question time at last - and I asked how did they get the administrators etc to engage with defining the data and processes etc - the answer:
- it took a year and a half!
- the process was iterative - defining requirements of what you actually want to do and what data is needed - defining use cases (I'm assuming UML use cases?), implementing, testing and piloting some of the service layer then going back round the loop again
- looking for commonality of data elements and processes across all the requirements
In a nutshell slowly and with a LOT of effort. Mmmm - this is going to be a tough project for us but one we MUST do if we are going to deliver our corporate strategy.
We need to deliver some solid results by the beginning of next academic year and we're not going to be able to do everything by then - sounds like DSDM might be a good way of tackling this.
Another question was asked from the floor about the difficulty of defining access/permissions. Response was that this took a lot of time too - and when you think about it that's not surprising. You are no longer defining access/permissions to individual databases that are 'owned' by seperate groups of users (or organisational silos). Instead you are having to identify access/permission to a common entity (the service layer) for all groups. It could be complicated and would almost certainly mean changes to the current access requirements to simplify and consolidate them.
Looks like that's it - so the take-away (as our American colleagues say). Implementing SOA is going to be just as hard as we thought it would be, and probably harder - but the benefits are more than worth it and in fact are probably essential for organisational survival given the exponential rise in the volume and pace of changing requirements.
Next stop - coffee!!