Scipio ERP 3.0.0 Released

by | Dec 23, 2023 | Release | 0 comments

Scipio ERP 3.0.0 has been released! This version represents an evolution of framework language and API enhancements meant to improve extensibility and reuse of framework components by client code and developers, as well as coding agility and facility. A large number of improvements to rendering utilities and application responsiveness are also available, as well as various new addons, theme and other additions.

Theme enhancements

A new Bulma theme is now the default for Scipio ERP, which sports a dynamic menu presentation over traditional themes.

New Enterprise Addons

Java JDK support (11/17)

This version officially drops support for Java 8 (with its -source/-target compiler flags) in favor of Java 11 (LTS), which is the default target for compiled code. JDK 17 (LTS) can also be used with the main platform, but due to the variety of third-party libraries still in use, compatibility is better maintained by targeting 11 by default.

To override the default target (which now uses the Java 9+ ‘-release’ argument), either modify build.properties#scipio.build.javac.release=11/17 for your derived project or create a file named build.scp.local.properties in your project root having scipio.build.javac.release=11/17. *.scp.local.properties files are ignored by git and are meant for individual developers and per-machine deployments. Due to Java 15’s removal of the built-in Nashorn JS engine, it is now automatically downloaded if this version is used (if this does not work, try ./ant download-ant-js manually).

IntelliJ Plugin

A new version of our Scipio ERP integration for IntelliJ IDEA has made its way to the JetBrains marketplace. The new release features:

  • Support for IntelliJ 2023.*
  • Improved Reload Resources performance
  • Improved Reload Resources jarDirectory/root duplicate detection
  • Improved service location reference resolution (java, groovy, simple)
  • Improved controller event (java, simple), response and view-map reference resolution and integration

The plugin remains the most convenient way to develop Scipio and we highly recommend it to anyone.

Service Engine

New Service Definitions (@Service/LocalService)

Services – the platform’s most essential building blocks – can now be defined using the @Service Java annotation and implemented using a dynamic and extensible class-based implementation using the abstract facade class LocalService (ServiceHandler). Together they help cement Java as the frequently strongest choice to implement solid, reliable, well-checked and extensible services.

@Service

@Service annotations are used inline to annotate Java service implementations as a complete replacement for traditional XML-based services*.xml files, making it is easier to create and maintain new services. Both methods are still supported and fully interchangeable, and @Service works on both traditional static method and new class-based service implementations. @Service annotations can also be accompanied by the new @Seca and @Eeca service annotations, which substitute for traditional XML-based service- and entity- ECA events, which are logically designed to annotate the target services they invoke (not the service they are intercepting, in the case of @Seca).

LocalService (ServiceHandler)

Class-based service implementations are small classes implementing ServiceHandler that invoke a variant of an exec() method, as a replacement for traditional static methods that typically have to be copy-pasted. We typically defined them as nested classes within existing *Services.java implementation container files. When invoked from legacy XML service definitions, the location attribute is typically a nested class and the invoke attribute simply specifies exec(): location="org.ofbiz.common.CommonServices$EchoService" invoke="exec". However, they excel most when combined with annotations.

ServiceHandler interfaces support various modes of invocation. However, repeated usage and analysis has shown the best invocation method to be the one recommended for use and implemented by the LocalService abstract facade class, which is a per-invocation instantiation with dedicated attribute-processing init(ServiceContext) method, with suggestion of using protected fields for holding attribute values and providing the standard ctx/dctx/context variables using protected fields, in short. Further enhancing it are the new classes, ServiceContext and ServiceResult. ServiceContext combined DispatchContext, the context map, utility methods and contextual information; ServiceResult removes the need for ServiceUtil using success()/fail()/error() and other utility methods.

This pattern provides maximum flexibility to client code that needs to extend and reuse framework services and its own services, and permits both easy translation of traditional static method-based services into class-based ones and a writing style highly reminiscent of writing in a script language (but without overhead and better IDE-based, compile-time error detection). Using protected fields helps promote script-like code simplicity and low upkeep, and are made manageable by a dedicated init(ServiceContext) method that allows sub-classes to have full control over their initialization (as opposed to using an unavoidable constructor).

Instead of writing boilerplate getters (which complicates migration and was rarely done in script languages that used local variable bindings), it is encouraged for developers to split their exec() method into logical units they expect client code and specializations will want to override. This way, the amount of code duplication between projects is kept to a minimum, and service code reuse is maximized. Note that class-based implementations can still be defined using XML service interface definitions, such that @Service and LocalService are fully decoupled, making code migration easy.

Here is an example from ServiceTestServices (service component), which showcases usage examples related to @Seca, @Eeca, variants of attribute processing and injection and the various supported patterns for versatility and corner cases:


    @Service
    @Attribute(name = "param1", type = "String", mode = "IN", defaultValue = "test value 1", optional = "true")
    @Attribute(name = "param2", type = "String", mode = "IN", defaultValue = "test value 2", optional = "true")
    @Attribute(name = "result1", type = "String", mode = "OUT", optional = "true")
    @PermissionService(service = "commonGenericPermission", mainAction = "UPDATE")
    @Permission(permission = "ENTITYMAINT")
    public static class ServiceAnnotationsTest2 extends LocalService {
        protected static final Debug.OfbizLogger module = Debug.getOfbizLogger(java.lang.invoke.MethodHandles.lookup().lookupClass());

        protected String param1;
        protected String param2; // NOTE: This shorthand is less efficient for

        @Override
        public void init(ServiceContext ctx) throws GeneralException {
            super.init(initServiceLogNew(ctx, module)); // Allows subclasses to specify srvModule
            param1 = ctx.attr("param1");
            param2 = ctx.attr("param2");
        }

        @Override
        public Map exec() {
            Debug.logInfo(ctx.serviceName() + ": input: " + ctx.context(), srvModule);
            Map result = ctx.success();
            result.put("result1", "test result value 1; concatenated param values: " + concatenateParams());
            return result;
        }

        protected String concatenateParams() {
            return param1 + "::" + param2;
        }
    }

    @Service(
            implemented = {
                    @Implements(service = "serviceAnnotationsTest2")
            },
            attributes = {
                    @Attribute(name = "param3", type = "String", mode = "IN", defaultValue = "test value 3", optional = "true")
            },
            auth = "true",
            permissionService = {
                    @PermissionService(service = "commonGenericPermission", mainAction = "UPDATE")
            }
    )
    public static class ServiceAnnotationsTest3 extends ServiceAnnotationsTest2 {
        protected static final Debug.OfbizLogger module = Debug.getOfbizLogger(java.lang.invoke.MethodHandles.lookup().lookupClass());
        protected String param3;

        @Override
        public void init(ServiceContext ctx) throws GeneralException {
            super.init(initServiceLogNew(ctx, module));
            param3 = ctx.attr("param3");
        }

        @Override
        protected String concatenateParams() {
            return super.concatenateParams() + "::" + param3;
        }
    }

These have already been used frequently throughout our projects. Here is a real-world example from OrderServices:


    @Service(
            description = "Populates a best-selling category with products based on sales total or quantity ordered"
    )
    @Attribute(name = "productCategoryId", type = "String", mode = "IN", optional = "false")
    @Attribute(name = "orderByType", type = "String", mode = "IN", optional = "true", defaultValue = "quantity-ordered",
            description = "Ordering criteria, one of (extensible by overriding services): " +
                    "sales-total: use BestSellingProductsBySalesTotal view-entity or equivalent; " +
                    "quantity-ordered: use BestSellingProductsByQuantityOrdered view-entity or equivalent; " +
                    "order-item-count: use BestSellingProductsByQuantityOrdered view-entity or equivalent")
    @Attribute(name = "productStoreIdList", type = "List", mode = "IN", optional = "true",
            description = "Filter for OrderHeader.productStoreId")
    @Attribute(name = "productStoreId", type = "String", mode = "IN", optional = "true",
            description = "Filter for OrderHeader.productStoreId")
    @Attribute(name = "filterCategoryId", type = "String", mode = "IN", optional = "true",
            description = "Only products belonging to this category and sold from this category are considered, using OrderItem.productCategoryId")
    @Attribute(name = "filterCategoryIdWithParents", type = "String", mode = "IN", optional = "true",
            description = "Only products belonging to this category or whose (virtual) parents are in the category or are considered")
    @Attribute(name = "filterSalesDiscDate", type = "Boolean", mode = "IN", optional = "true", defaultValue = "true")
    @Attribute(name = "maxProducts", type = "Integer", mode = "IN", optional = "true")
    @Attribute(name = "orderDateStart", type = "Timestamp", mode = "IN", optional = "true",
            description = "Absolute start date if specified")
    @Attribute(name = "orderDateEnd", type = "Timestamp", mode = "IN", optional = "true",
            description = "Absolute end date if specified; defaults to now")
    @Attribute(name = "orderDateDays", type = "Integer", mode = "IN", optional = "true", defaultValue = "30",
            description = "Days to substract from orderDateEnd to get absolute start date")
    @Attribute(name = "updateMode", type = "String", mode = "IN", optional = "true", defaultValue = "update",
            description = "Update mode, one of: " +
                    "update: update records in-place where necessary and possible; " +
                    "create: always delete and recreate records (makes records sequenced by date)")
    @Attribute(name = "removeMode", type = "String", mode = "IN", optional = "true", defaultValue = "remove",
            description = "Removal mode, one of: " +
                    "remove: remove old records; " +
                    "preserve: ignore old records (faster)")
    @Attribute(name = "logEvery", type = "Integer", mode = "IN", optional = "true", defaultValue = "50")
    public static class PopulateBestSellingCategory extends LocalService {
        protected static final Debug.OfbizLogger module = Debug.getOfbizLogger(java.lang.invoke.MethodHandles.lookup().lookupClass());

        protected String productCategoryId;
        protected GenericValue productCategory;
        protected int productCount = 0;
        protected int removed = 0;
        protected int created = 0;
        protected int updated = 0;
        protected int maxTopProductIds = 10;
        protected Set topProductIds = new LinkedHashSet<>();
        protected Collection productStoreIds = null;
        protected long sequenceNum = 1;
        protected Timestamp orderDateStart;
        protected Timestamp orderDateEnd;
        protected Integer orderDateDays;
        protected String filterCategoryId;
        protected String filterCategoryIdWithParents;
        protected Timestamp nowTimestamp;
        protected boolean removeOld;
        protected boolean createNew;
        protected boolean filterSalesDiscDate;
        protected Integer logEvery;

        @Override
        public void init(ServiceContext ctx) throws GeneralException {
            super.init(initServiceLogNew(ctx, module));
            if (nowTimestamp == null) {
                nowTimestamp = UtilDateTime.nowTimestamp();
            }
            productCategoryId = ctx.attr("productCategoryId");
            productStoreIds = ctx.attrNonEmpty("productStoreIdList");
            String productStoreId = ctx.getStringNonEmpty("productStoreId");
            if (UtilValidate.isNotEmpty(productStoreId)) {
                Set newProductStoreIds = new LinkedHashSet<>();
                newProductStoreIds.add(productStoreId);
                if (UtilValidate.isNotEmpty(productStoreIds)) {
                    newProductStoreIds.addAll(productStoreIds);
                }
                productStoreIds = newProductStoreIds;
            }
            orderDateStart = ctx.attr("orderDateStart");
            orderDateEnd = ctx.attr("orderDateEnd"); // don't force here: UtilDateTime::nowTimestamp
            orderDateDays = ctx.attr("orderDateDays");
            if (orderDateStart == null && orderDateDays != null && orderDateDays > 0) {
                if (orderDateEnd == null) {
                    orderDateEnd = nowTimestamp;
                }
                orderDateStart = UtilDateTime.addDaysToTimestamp(orderDateEnd, -orderDateDays);
            }
            filterCategoryId = ctx.attr("filterCategoryId");
            filterCategoryIdWithParents = ctx.attr("filterCategoryIdWithParents");
            removeOld = "remove".equals(ctx.attr("removeMode"));
            createNew = "create".equals(ctx.attr("updateMode"));
            filterSalesDiscDate = ctx.attr("filterSalesDiscDate", true);
            logEvery = ctx.attr("logEvery");
        }

        @Override
        public Map exec() throws ServiceValidationException {
            try {
                productCategory = ctx.delegator().from("ProductCategory").where("productCategoryId", productCategoryId).queryOne();
            } catch (GenericEntityException e) {
                Debug.logError(e, srvModule);
                return ServiceUtil.returnError(e.toString());
            }
            if (productCategory == null) {
                return ServiceUtil.returnError("ProductCategory [" + productCategoryId + "] not found");
            }

            Set unseenProductIds = null;
            if (removeOld) {
                if (createNew) {
                    // Remove old immediately
                    Iterator catIt = null;
                    try {
                        // Might not trigger all needed ECAs, so do one-by one...
                        //int removed = ctx.delegator().removeByAnd("ProductCategoryMember", "productCategoryId", productCategoryId);
                        catIt = UtilMisc.asIterator(getCategoryMembers(productCategoryId));
                        GenericValue pcm;
                        while((pcm = UtilMisc.next(catIt)) != null) {
                            removeProductCategoryMember(pcm);
                        }
                        Debug.logInfo("Removed " + removed + " old ProductCategoryMember records for category [" +
                                productCategoryId + "]", srvModule);
                    } catch (GeneralException e) {
                        Debug.logError(e, srvModule);
                        return ServiceUtil.returnError(e.toString());
                    } finally {
                        if (catIt instanceof AutoCloseable) {
                            try {
                                ((AutoCloseable) catIt).close();
                            } catch (Exception e) {
                                Debug.logError(e, srvModule);
                            }
                        }
                    }
                } else {
                    // Delayed remove
                    try {
                        unseenProductIds = getCategoryProductIdSet(productCategoryId);
                    } catch (GeneralException e) {
                        Debug.logError(e, srvModule);
                        return ServiceUtil.returnError(e.toString());
                    }
                }
            }

            Iterator> prodIt = null;
            try {
                Debug.logInfo("Beginning query on category [" + productCategoryId + "]", srvModule);
                prodIt = UtilMisc.asIterator(getProducts());
                Integer maxProducts = ctx.attr("maxProducts");
                Map productEntry;
                while (((productEntry = UtilMisc.next(prodIt)) != null) && (maxProducts == null || productCount < maxProducts)) {
                    String productId = (String) productEntry.get("productId");
                    ProductInfo info = makeProductInfo(productEntry, sequenceNum);
                    if (!useProduct(info)) {
                        continue;
                    }
                    GenericValue pcm = createUpdateProductCategoryMember(info);
                    if (unseenProductIds != null) {
                        unseenProductIds.remove(productId);
                    }
                    sequenceNum++;
                    if (pcm != null) {
                        productCount++;
                    }
                    if (logEvery != null && (sequenceNum % logEvery == 0)) {
                        Debug.logInfo("Visited " + sequenceNum + " records, added " + productCount + " products", srvModule);
                    }
                }
            } catch(GeneralException e) {
                Debug.logError(e, srvModule);
                return ServiceUtil.returnError(e.toString());
            } finally {
                if (prodIt instanceof AutoCloseable) {
                    try {
                        ((AutoCloseable) prodIt).close();
                    } catch (Exception e) {
                        Debug.logError(e, srvModule);
                    }
                }
            }

            if (removeOld && UtilValidate.isNotEmpty(unseenProductIds)) {
                Iterator catIt = null;
                try {
                    catIt = UtilMisc.asIterator(getCategoryMembers(productCategoryId, unseenProductIds));
                    GenericValue pcm;
                    while((pcm = UtilMisc.next(catIt)) != null) {
                        removeProductCategoryMember(pcm);
                    }
                } catch (GeneralException e) {
                    Debug.logError(e, srvModule);
                    return ServiceUtil.returnError(e.toString());
                } finally {
                    if (catIt instanceof AutoCloseable) {
                        try {
                            ((AutoCloseable) catIt).close();
                        } catch (Exception e) {
                            Debug.logError(e, srvModule);
                        }
                    }
                }
            }

            String msg = getSuccessMsg();
            Debug.logInfo("populateBestSellingCategory: " + msg, srvModule);
            return ServiceUtil.returnSuccess(msg);
        }

        /**
         * Returns iterator or collection of Map product entries.
         */
        protected Object getProducts() throws GeneralException {
            String orderByType = ctx.attr("orderByType");
            if ("quantity-ordered".equals(orderByType)) {
                return getProductsByQuantityOrdered();
            } else if ("sales-total".equals(orderByType)) {
                return getProductsBySalesTotal();
            } else if ("order-item-count".equals(orderByType)) {
                return getProductsByOrderItemCount();
            //} else if ("solr".equals(orderByType)) {
            //    return getProductsBySolr();
            } else {
                throw new ServiceValidationException("Invalid orderByType", ctx.getModelService());
            }
        }

        protected Object getProductsBySalesTotal() throws GeneralException {
            DynamicViewEntity dve = ctx.delegator().makeDynamicViewEntity("BestSellingProductsBySalesTotal");
            EntityCondition cond = makeCommonCondition();
            cond = addFilterCategoryIdCond(cond, dve);
            return ctx.delegator().from(dve).where(cond).orderBy("-salesTotal").cache(false).queryIterator();
        }

        protected Object getProductsByQuantityOrdered() throws GeneralException {
            DynamicViewEntity dve = ctx.delegator().makeDynamicViewEntity("BestSellingProductsByQuantityOrdered");
            EntityCondition cond = makeCommonCondition();
            cond = addFilterCategoryIdCond(cond, dve);
            return ctx.delegator().from(dve).where(cond).orderBy("-quantityOrdered").cache(false).queryIterator();
        }

        protected Object getProductsByOrderItemCount() throws GeneralException {
            DynamicViewEntity dve = ctx.delegator().makeDynamicViewEntity("BestSellingProductsByOrderItemCount");
            EntityCondition cond = makeCommonCondition();
            cond = addFilterCategoryIdCond(cond, dve);
            return ctx.delegator().from(dve).where(cond).orderBy("-orderItemCount").cache(false).queryIterator();
        }

        protected EntityCondition makeCommonCondition() throws GeneralException {
            EntityCondition cond = EntityCondition.makeDateRangeCondition("orderDate", orderDateStart, orderDateEnd);
            if (UtilValidate.isNotEmpty(productStoreIds)) {
                cond = EntityCondition.combine(cond,
                        EntityCondition.makeCondition("productStoreId", EntityOperator.IN, productStoreIds));
            }
            return cond;
        }

        protected EntityCondition addFilterCategoryIdCond(EntityCondition cond, DynamicViewEntity dve) {
            if (filterCategoryId != null) {
                if (dve != null) {
                    dve.addMemberEntity("PCM", "ProductCategoryMember");
                    dve.addAlias("PCM", "filterCategoryId", "productCategoryId", false, null, false);
                    dve.addViewLink("OI", "PCM", false,
                            UtilMisc.toList(new ModelKeyMap("productId", "productId")));
                }
                EntityCondition filterCond = EntityCondition.makeCondition("filterCategoryId", filterCategoryId);
                cond = EntityCondition.makeCondition(cond, EntityOperator.AND, filterCond);
            }
            return cond;
        }

        protected Object getCategoryMembers(String productCategoryId) throws GeneralException {
            return ctx.delegator().from("ProductCategoryMember").where("productCategoryId", productCategoryId)
                    .queryIterator();
        }

        protected Set getCategoryProductIdSet(String productCategoryId) throws GeneralException {
            return ctx.delegator().from("ProductCategoryMember").where("productCategoryId", productCategoryId)
                    .getFieldSet("productId");
        }

        protected Object getCategoryMembers(String productCategoryId, Collection productIdList) throws GeneralException {
            return ctx.delegator().from("ProductCategoryMember").where(
                    EntityCondition.makeCondition("productCategoryId", productCategoryId),
                    EntityCondition.makeCondition("productId", EntityOperator.IN, productIdList))
                    .queryIterator();
        }

        /**
         * Returns null if none created/skip.
         */
        protected GenericValue createUpdateProductCategoryMember(ProductInfo info) throws GeneralException {
            List prevPcms = ctx.delegator().from("ProductCategoryMember")
                    .where("productCategoryId", productCategoryId, "productId", info.getProductId())
                    .filterByDate(nowTimestamp).queryList();
            GenericValue pcm;
            if (UtilValidate.isNotEmpty(prevPcms)) {
                if (prevPcms.size() > 1) {
                    Debug.logWarning("ProductCategoryMember productCategoryId [" + productCategoryId +
                            "] productId [" + info.getProductId() + "] has more than one (" + prevPcms.size() +
                            ") entry for category; updating first only", srvModule);
                }
                pcm = prevPcms.get(0);
                pcm.set("sequenceNum", info.getSequenceNum());
                pcm.store();
                updated++;
            } else {
                pcm = ctx.delegator().makeValue("ProductCategoryMember",
                        "productCategoryId", productCategoryId,
                        "productId", info.getProductId(),
                        "fromDate", nowTimestamp,
                        "sequenceNum", info.getSequenceNum()).create();
                created++;
            }
            if (topProductIds.size() < maxTopProductIds) {
                topProductIds.add(info.getProductId());
            }
            return pcm;
        }

        /** Applies individual product filters; may be overridden. */
        protected boolean useProduct(ProductInfo info) throws GeneralException {
            if (filterCategoryIdWithParents != null && !hasCategoryIdWithParents(info)) {
                return false;
            }
            if (filterSalesDiscDate && !isOkSalesDiscDate(info)) {
                return false;
            }
            return true;
        }

        protected boolean hasCategoryIdWithParents(ProductInfo info) throws GeneralException {
            return hasCategoryIdWithParents(info.getProductId(), info.getProduct(), filterCategoryIdWithParents);
        }

        protected boolean hasCategoryIdWithParents(String productId, Map product,
                                                   String filterCatId) throws GeneralException {
            if (ctx.delegator().from("ProductCategoryMember")
                    .where("productCategoryId", filterCatId, "productId", productId)
                    .filterByDate(nowTimestamp).queryCount() > 0) {
                return true;
            }
            if (product == null) {
                product = ctx.delegator().from("Product").where("productId", productId).queryOne();
                if (product == null) {
                    Debug.logError("Could not find product [" + productId + "]", srvModule);
                    return false;
                }
            }
            String parentProductId = getParentProductId(productId, product);
            if (parentProductId != null) {
                return hasCategoryIdWithParents(parentProductId, null, filterCatId);
            }
            return false;
        }

        protected boolean isOkSalesDiscDate(ProductInfo info) throws GeneralException {
            Timestamp salesDiscontinuationDate = info.getProduct().getTimestamp("salesDiscontinuationDate");
            if (salesDiscontinuationDate != null) {
                return !salesDiscontinuationDate.before(nowTimestamp);
            };
            return true;
        }

        protected String getParentProductId(String productId, Map product) throws GeneralException {
            Object isVariant = product.get("isVariant");
            if ((isVariant instanceof Boolean && ((Boolean) isVariant)) || "Y".equals(isVariant)) {
                GenericValue parentProductAssoc = ProductWorker.getParentProductAssoc(productId, ctx.delegator(),
                        nowTimestamp, false);
                if (parentProductAssoc != null) {
                    return parentProductAssoc.getString("productId");
                }
            }
            return null;
        }

        protected void removeProductCategoryMember(GenericValue pcm) throws GeneralException {
            pcm.remove();
            removed++;
        }

        protected String getSuccessMsg() {
            return "Category: " + productCategoryId + ": " + productCount + " products found; " + created +
                    " members created; " + updated + " members updated; " + removed +
                    " members removed or recreated; top products: " + topProductIds;
        }

        protected ProductInfo makeProductInfo(Map productEntry, long sequenceNum) {
            return new ProductInfo(productEntry, sequenceNum);
        }

        protected class ProductInfo {
            /** Usually a view-entity value, NOT Product */
            protected Map productEntry;
            protected String productId;
            protected long sequenceNum;
            protected GenericValue product;

            protected ProductInfo(Map productEntry, long sequenceNum) {
                this.productEntry = productEntry;
                this.productId = (String) productEntry.get("productId");
                this.sequenceNum = sequenceNum;
            }

            public Map getProductEntry() {
                return productEntry;
            }

            public String getProductId() {
                return productId;
            }

            public long getSequenceNum() {
                return sequenceNum;
            }

            public GenericValue getProduct() throws GenericEntityException {
                GenericValue product = this.product;
                if (product == null) {
                    product = ctx.delegator().from("Product").where("productId", getProductId()).queryOne();
                    this.product = product;
                }
                return product;
            }
        }
    }

Dispatcher API

LocalDispatcher has been enhanced and re-standardized for pattern parameter passing to runAsync() methods (AsyncOptions), better job control return (JobInfo), better factory/accessor/facade methods (LocalDispatcher.from()/fromName()/getDefault()), more makeValidContext() method overloads, and a slew of other and internal improvements.

Job Manager

Improvements to statistics, job ID returning (Dispatcher), job queueing strategies, crashed job handling, Admin UI screens and more.

Entity Engine

The Delegator, GenericEntity and GenericValue interfaces and classes have been largely improved.

Delegator API

Delegator interface has been enhanced with better factory/accessor/facade methods (Delegator.from()/fromName()/getDefault()), so there is no need to reference DelegatorFactory or others anymore. EntityQuery can be gotten directly from Delegator using a slew of new facade methods, deprecating the need for redundant helpers everywhere else: delegator.query().from("Product").where("productId", productId).cache(useCache).queryOne() or delegator.from("Product").where("productId", productId).cache(useCache).queryOne(). These are already in frequent use throughout the source and have been formalized.

JSON Storage

Automatic JSON field parsing, conversion, storage and caching is implemented via GenericEntity, through getJson()/setJson() methods. JSON serialized in database as text fields are automatically unpickled to Java Map and List types. The JSON read cache is on by default and invalidated when the field is modified. It is efficient and able to handle frontend use by default. Entity fields storing JSON must be marked either very-long (legacy) or one of the 3 new types: json-object, json-array, json. Marking the specific type may allow better use and type safety in the future. By convention, entities may wish to define a generic JSON field named entityJson (of json-object type), which is read by getEntityJson()/setEntityJson() convenience methods; this is wanted to extend entity in a generic way without having to define more fields, since they can be stored in JSON instead (at the expense of SQL querying).

Entity and Solr Indexing

  • Indexing: Solr ECA-based indexing was enhanced for high-performance updates using and refining an efficient global queue system, implemented through EntityIndexer, and generified to produce documents in a variety of format for different target commercial platforms. Product entity modifications that traditionally called addToSolr/updateToSolr now call scheduleProductIndexing, which schedules a Solr document update for the product only at the end of a successful transaction in which they’re modified, which is done transparently by the service (not needed to be indicated on the ECA definition; see applications/solr/entitydef/eecas.xml). The queue pipeline eliminates duplicates within a timeframe and allows for client-code based transformations by extending SolrDocBuilder.ProductDocBuilder through factory properties configuration. At the end of the pipeline, configurable productIndexingConsumer service implementations accept the produced Solr documents, convert them and send them to the target platform – typically now optimized using connection pooling. These implementations are configured in entityindexing.properties files.
  • Schema: The Solr schema has largely been migrated to dynamic field usage and now indexes with better currency precision types; however some legacy static fields are still in use.

Excel import/export

A configurable (properties-based) Excel sheet entity and Product importer/exporter is now built-in under the Admin component, under Import/Export.

Webserver

WebSocket Support

WebSockets – provided by Scipio’s webserver and servlet container – allow a push-model communication from server to client, as opposed to traditional client to server polling/refresh strategies. This allows for highly responsive and interaction applications. These exchanges are normally done in JSON.

Scipio 3.0.0 improves upon and standardizes session and event management classes to more easily and safely use them within the platform and in new applications. SocketSessionManager was rewritten. See GenericWebSocket/CommonWebSocket and the examples already strewn across the various components: OrderWebSocket, AdminWebSocket, etc.

The servlet container now automatically whitelist WebSockets URI request paths defined within component Java sources using the standard @ServerEndpoint annotation – this was traditionally done by editing allowed paths under web.xml, which is no longer required. See OrderWebSocket and the Admin layout demo page WebSockets section for example.

Turbo JS/HotWire

Included are new usage patterns enabling the use of the Turbo JS/Hotwire framework, which which does away with Javascript-heavy application development. Hotwire makes this integration with Scipio natural. Hotwire prominently makes use of Websockets to achieve interactive applications, which are encapsulate for easy use within Scipio as TurboWebSocket, with AdminTurboWebSocket as prominent example. Head over to the Admin layout demo page Hotwire section for examples to get started.

Controller and Event

  • @Controller/@Request/@View: A new Java @Request annotation has been added to help define controller request-map URIs for Java static event methods with no need to edit controller.xml files. @Controller(controller = "xxx") is used to annotate the containing class file and indicates the target controller for all the @Request definitions contained. @View can be used to annotate a static field to define a view without using XML; typically it will annotate a field defining a view control response (ControlResponse.View), though not strictly required. @Request-annotated events have a different interface than their traditional XML-invoked counterparts: instead of returning only response strings, they return the entire response, one of the implementations of the ControlResponse class, such as ControlResponse.View or ControlResponse.ViewLast. This allows more powerful response and view-selection logic in an easier form than traditional XML-based controller responses, and in fact allows to fully customize control logic, which was sometimes a limiting factor in legacy code. All of these annotations are fully integrated into the controller system. See AdminController for example.
  • Inline scripts: Controller events now support inline groovy scripts in XML (using CDATA).
  • Security: RequestHandler/ContextFilter: There is no more dumping of last-view request parameters directly into request attributes (instead merged through UtilHttp.getParameterMap() and the like); client code can filter allowed request attributes from parameters using a plugin system (RequestAttrPolicy); returned JSON responses are filtered to exclude more request attributes; improved code quality through refactors; various more.

CMS

This release adds Robots.txt configuration support, better WISIWYG editor integration, Redirect Management, Cache Prewarming and other features to Scipio’s built-in content management system. Image variant generation improvements are also inlined into the image upload.

Rendering API

  • Image variants/resizing: Automatic image variant creation and resizing has been improved and made further configurable using mediaprofiles.properties/ImageProperties.xml files as well as through CMS’s dynamic entities (ImageSizePreset) and through Content/DataResource records. Resizing is implemented using contentImageAutoRescale/productImageAutoRescale core services, which are invoked by uploading images to CMS in the backend as well as applied to data in batches using productImageAutoRescaleProducts/productImageAutoRescaleAll. Auto-rescaling can now be triggered automatically and ensured when pages are rendered that contain links to them. This is done using the new ProductImageVariants/CategoryImageVariants helper classes, which supersede the use of ProductContentWrapper for querying variants. In other words, the following Freemarker code will obtain an info class for the “large” (LARGE_IMAGE_URL/Product.largeImageUrl) variant, and if configured to do so, will trigger a background automatic image rescaling for the queried image if it does not already exist: (productContentWrapper.getImageVariants("ORIGINAL_IMAGE_URL").getVariant("large").getImageUrl(context))!
  • Image optimizations: Tinify now supported to optimize images for size and download cost; configure using imageops.properties.
  • Caching: Freemarker @utilCache macro is now used throughout templates to support caching specific template parts; FreeMarkerWorker *.ftl template caching improved using standard Configuration mechanisms.
  • javaScriptEnabled: The ancient “Y”/”N” indicator flag was used inconsistently and is now largely ignored or removed.

Localized Resource Properties

  • Global Resource Bundle: Localized resource bundles (*Labels.xml) used to localized messages in screens, Freemarker templates, services and framework error messages are now fully global by default and no longer require to specify a specific “resource” file in the majority of cases, requiring only the property name (it can be left an empty string or you can use an overload without “resource”). This decision was made to greatly simplify and optimize resource bundle loading and usage throughout the platform, but especially because it allows hot-deploy/client component to easily and automatically override labels define in framework and application components, platform-wide, since they now find themselves in a single global bundle. This works and was acceptable because the large majority of label names were already globally unique, similar to the global service namespace. On first access, all the resource bundles in all components matching the config/*Labels.xml file naming pattern are combined into a single global bundle, which is nearly always returned by UtilProperties.getMessage() – even if an explicit resource file is requested (unless the property name is not found within, in which case the passed resource file is checked as fallback).

Core and Utility API

A large number of API improvements concerning Java reflection and annotations, theming, Freemarker-based rendering (utilities.ftl and others) have been added.

  • Connections: ScipioHttpClient is now used for synchronous and asynchronous requests to both internal and external addresses to promote connection pooling and reuse. It makes property-based configurations easy for any component needing it. Many addons now make use of it.
  • Exception handling: Most Scipio classes now extend GenericException or GeneralRuntimeException, which have been augmented with support for user-facing localized property-based error messages. Exceptions can more easily define an internal/logging English message while also carrying friendly localized messages to return to users from services and events. GeneralException/GenericRuntimeException are both outfitted with a propertyMessageList field – an explicitly “additional” message list, separate from both the main detail message and the propertyMessage field – the list being being specifically intended for user-facing messages although it can be used for other purposes to your discretion. The exception constructors have been standardized to reflect this, but existing exception hierarchies can be bypassed using the setPropertyMessageList() method. Note: Contrarily, use of setPropertyMessage() is discouraged as it reflects an override of the main internal/logging exception detail message, not the additional (user) messages list – you should almost always specify a main exception detail message in addition to and separately from the additional (user) messages list.
  • UtilCache: Optimized and streamlined to remove unnecessary contention from execution pulses (which had significant impacts in real-world scenarios) and synchronized erases and removals; other improvements. Some additional features have been held back to maximize speed, simplicity and non-locking behavior of ConcurrentHashMap.
  • Distributed cache clearing: Several services newly implement ActiveMQ-based distributed cache clearing, such as image variants.
  • Java annotations: A new ReflectQuery API is used to scan component Java sources and make them available to webapp- and platform-related code; annotations are generally per-component but are queryable by webapp.

Build

  • New helper build targets: load-extseedonly, rebuild, rebuild-debug, restart, restart-debug, etc.
  • Build properties: build.properties values are now passed everywhere and preempt those from macros.xml, including useful javac values such as scipio.build.java.release (which corresponds to -release; -source/-target are removed for JDK 11 standardization). build.scp.local.properties can be created in project root and contain any arbitrary global build properties for individual developers and machines. *.scp.local.properties files are ignored by git.

Shop, Order, Products

  • Locale/TimeZone: Initial setting of session locale, timeZone and currencyUom has been revamped for perfect consistency using AttrHandler. Also applies to backend applications.
  • ShoppingList: Now uses an authToken for security; other improvements.
  • ShoppingCart: Numerous cart additions; improved polymorphism.
  • Sitemap: Sitemap is now fully localized with improved output, CMS/content pages and other features.
  • Promotions: Several fixes and improvements.
  • calculateProductPrice: Price calculations have been improved to handle more complex variant cases, tax/VAT handling and better integrate with Solr/entity indexing.
  • Cookies: Better standardized and guarded against unauthorized characters using URL encoding; some cookies in old forms will be recreated at first user visit or login.
  • URL building: Product ALTERNATIVE_URL generation improved.
  • Solr/Product search: Improved searching and result caching code and configuration for Solr-based queries in frontend.

Library Updates

Numerous third-party libraries have been updated. Notably, Apache fileupload has a recent fix for a long-standing bug, and several new libraries have been added.

Fixes

Many fixes have been applied to framework and business application logic.